{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tutorial on Hyperparameter Tuning in Neural Networks\n",
    "\n",
    "**Author:** Matthew Stewart\n",
    "<br>\n",
    "\n",
    "This notebook follows the same procedure as the Medium article **\"Simple Guide to Hyperparameter Tuning in Neural Networks\"**.\n",
    "\n",
    "[https://medium.com/@matthew_stewart/simple-guide-to-hyperparameter-tuning-in-neural-networks-3fe03dad8594]. \n",
    "\n",
    "In this notebook we will optimize and fine tune a neural network to find the global minimum of a particularly troublesome function known as the Beale function. This is one of many test functions for optimization that are commonly used in academia (see https://en.wikipedia.org/wiki/Test_functions_for_optimization for more information).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "code_folding": [
     0
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>\n",
       "blockquote { background: #AEDE94; }\n",
       "h1 { \n",
       "    padding-top: 25px;\n",
       "    padding-bottom: 25px;\n",
       "    text-align: left; \n",
       "    padding-left: 10px;\n",
       "    background-color: #DDDDDD; \n",
       "    color: black;\n",
       "}\n",
       "h2 { \n",
       "    padding-top: 10px;\n",
       "    padding-bottom: 10px;\n",
       "    text-align: left; \n",
       "    padding-left: 5px;\n",
       "    background-color: #EEEEEE; \n",
       "    color: black;\n",
       "}\n",
       "\n",
       "div.exercise {\n",
       "\tbackground-color: #ffcccc;\n",
       "\tborder-color: #E9967A; \t\n",
       "\tborder-left: 5px solid #800080; \n",
       "\tpadding: 0.5em;\n",
       "}\n",
       "div.discussion {\n",
       "\tbackground-color: #ccffcc;\n",
       "\tborder-color: #88E97A;\n",
       "\tborder-left: 5px solid #0A8000; \n",
       "\tpadding: 0.5em;\n",
       "}\n",
       "div.theme {\n",
       "\tbackground-color: #DDDDDD;\n",
       "\tborder-color: #E9967A; \t\n",
       "\tborder-left: 5px solid #800080; \n",
       "\tpadding: 0.5em;\n",
       "\tfont-size: 18pt;\n",
       "}\n",
       "div.gc { \n",
       "\tbackground-color: #AEDE94;\n",
       "\tborder-color: #E9967A; \t \n",
       "\tborder-left: 5px solid #800080; \n",
       "\tpadding: 0.5em;\n",
       "\tfont-size: 12pt;\n",
       "}\n",
       "p.q1 { \n",
       "    padding-top: 5px;\n",
       "    padding-bottom: 5px;\n",
       "    text-align: left; \n",
       "    padding-left: 5px;\n",
       "    background-color: #EEEEEE; \n",
       "    color: black;\n",
       "}\n",
       "header {\n",
       "   padding-top: 35px;\n",
       "    padding-bottom: 35px;\n",
       "    text-align: left; \n",
       "    padding-left: 10px;\n",
       "    background-color: #DDDDDD; \n",
       "    color: black;\n",
       "}\n",
       "</style>\n",
       "\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Formatting\n",
    "import requests\n",
    "from IPython.core.display import HTML\n",
    "styles = requests.get(\"https://raw.githubusercontent.com/Harvard-IACS/2019-CS109B/master/content/styles/cs109.css\").text\n",
    "HTML(styles)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning Goals\n",
    "In this notebook, we will explore ways to optimize the loss function of a multilayer perceptor (MLP) by tuning the model hyperparameters. We will also explore the use of cross-validation as a technique for checking potential values for these hyperparameters.\n",
    "\n",
    "By the end of this notebook, you should:\n",
    "- Be familiar with the use of `sklearn`'s `optimize` function.\n",
    "- Be able to identify the hyperparameters that go into the training of a MLP.\n",
    "- Be familiar with the implementation in `keras` of various optimization techniques.\n",
    "- Know how to use callbacks\n",
    "- Apply cross-validation to check for multiple values of hyperparameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "from scipy.optimize import minimize\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1:  Beale's function\n",
    "### First let's look at function optimization in `scipy.optimize`, using Beale's function as an example\n",
    "\n",
    "Optimizing a function $f: A\\rightarrow R$, from some set A to the real numbers is finding an element $x_0\\,\\epsilon\\, A$ such that $f(x_0)\\leq f(x)$ for all $x\\,\\epsilon\\, A$ (finding the minimum) or such that $f(x_0)\\geq f(x)$ for all $x\\,\\epsilon\\, A$ (finding the maximum).\n",
    "\n",
    "To illustrate our point we will use a function of two parameters.  Our goal is to optimize over these 2 parameters. We can extend to higher dimensions by plotting pairs of parameters against each other.\n",
    "\n",
    "The Wikipedia article on Test functions for optimization has a few functions that are useful for evaluating optimization algorithms. Here is Beale's function:\n",
    "\n",
    "$f(x,y)$ = $(1.5−x+xy)^2+(2.25−x+xy^2)^2+(2.625−x+xy^3)^2$\n",
    "\n",
    "We already know that this function has a minimum at [3.0, 0.5]. Let's see if `scipy` will find it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![alt text](Bealesfunction.jpg)\n",
    "\n",
    "<pre>source: https://en.wikipedia.org/wiki/Test_functions_for_optimization</pre>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define Beale's function which we want to minimize\n",
    "def objective(X):\n",
    "    x = X[0]; y = X[1]\n",
    "    return (1.5 - x + x*y)**2 + (2.25 - x + x*y**2)**2 + (2.625 - x + x*y**3)**2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# function boundaries\n",
    "xmin, xmax, xstep = -4.5, 4.5, .9\n",
    "ymin, ymax, ystep = -4.5, 4.5, .9"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's create some points\n",
    "x1, y1 = np.meshgrid(np.arange(xmin, xmax + xstep, xstep), np.arange(ymin, ymax + ystep, ystep))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's make an initial guess"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "68891.203125\n"
     ]
    }
   ],
   "source": [
    "# initial guess\n",
    "x0 = [4., 4.]  \n",
    "f0 = objective(x0)\n",
    "print (f0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "bnds = ((xmin, xmax), (ymin, ymax))\n",
    "minimum = minimize(objective, x0, bounds=bnds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      fun: 2.068025638865627e-12\n",
      " hess_inv: <2x2 LbfgsInvHessProduct with dtype=float64>\n",
      "      jac: array([-1.55969780e-06,  9.89837957e-06])\n",
      "  message: b'CONVERGENCE: NORM_OF_PROJECTED_GRADIENT_<=_PGTOL'\n",
      "     nfev: 60\n",
      "      nit: 14\n",
      "   status: 0\n",
      "  success: True\n",
      "        x: array([3.00000257, 0.50000085])\n"
     ]
    }
   ],
   "source": [
    "print(minimum)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The answer, [3.00000257 0.50000085], is very close to the optimum as we know it, which is [3.0, 0.5]\n",
      "The value of the objective for [3.0, 0.5] is 0.0\n"
     ]
    }
   ],
   "source": [
    "real_min = [3.0, 0.5]\n",
    "print (f'The answer, {minimum.x}, is very close to the optimum as we know it, which is {real_min}') \n",
    "print (f'The value of the objective for {real_min} is {objective(real_min)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2: Optimization in neural networks\n",
    "\n",
    "In general: **Learning Representation --> Objective function --> Optimization algorithm**\n",
    "\n",
    "A neural network can be defined as a framework that combines inputs and tries to guess the output. If we are lucky enough to have some results, called \"the ground truth\", to compare the outputs produced by the network, we can calculate the **error**. So the network guesses, calculates some error function, guesses again, trying to minimize this error, guesses again, until the error does not go down any more. This is optimization.  \n",
    "\n",
    "In neural networks the most common used optimization algorithms, are flavors of **GD (gradient descent)**. The *objective function* used in gradient descent is the *loss function* which we want to minimize ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A `keras` Refresher"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Keras` is a Python library for deep learning that can run on top of both Theano or\n",
    "TensorFlow, two powerful Python libraries for fast numerical computing created and released by Facebook and Google, respectevely.\n",
    "\n",
    "Keras was developed to make developing deep learning models as fast and easy as\n",
    "possible for research and practical applications. It runs on Python 2.7 or 3.5 and can seamlessly execute on GPUs and CPUs.\n",
    "\n",
    "Keras is built on the idea of a model. At its core we have a sequence of layers called\n",
    "the `Sequential` model which is a linear stack of layers. Keras also provides the `functional API`, a way to define complex models, such as multi-output models, directed acyclic graphs, or models with shared layers.\n",
    "\n",
    "We can summarize the construction of deep learning models in Keras using the Sequential model as follows:\n",
    "1. **Define your model**: create a `Sequential` model and add layers.\n",
    "2. **Compile your model**: specify loss function and optimizers and call the `.compile()` function.\n",
    "3. **Fit your model**: train the model on data by calling the `.fit()` function.\n",
    "4. **Make predictions**: use the model to generate predictions on new data by calling functions such as `.evaluate()` or `.predict()`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Callbacks: taking a peek into our model while it's training\n",
    "\n",
    "You can look at what is happening in various stages of your model by using `callbacks`. A callback is a set of functions to be applied at given stages of the training procedure. You can use callbacks to get a view on internal states and statistics of the model during training. You can pass a list of callbacks (as the keyword argument callbacks) to the `.fit()` method of the Sequential or Model classes. The relevant methods of the callbacks will then be called at each stage of the training.\n",
    "\n",
    "- A callback function you are already familiar with is `keras.callbacks.History()`. This is automatically included in `.fit()`.\n",
    "\n",
    "\n",
    "- Another very useful one is `keras.callbacks.ModelCheckpoint` which saves the model with its weights at a certain point in the training. This can prove useful if your model is running for a long time and a system failure happens. Not all is lost then. It's a good practice to save the model weights only when an improvement is observed as measured by the `acc`, for example.\n",
    "\n",
    "\n",
    "- `keras.callbacks.EarlyStopping` stops the training when a monitored quantity has stopped improving.\n",
    "\n",
    "\n",
    "- `keras.callbacks.LearningRateScheduler` will change the learning rate during training.\n",
    "\n",
    "\n",
    "We will apply some callbacks later.\n",
    "\n",
    "For full documentation on `callbacks` see https://keras.io/callbacks/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What are the steps to optimizing our network?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.12.0\n",
      "2.1.6-tf\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "import keras\n",
    "from keras import layers\n",
    "from keras import models\n",
    "from keras import utils\n",
    "from keras.layers import Dense\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Flatten\n",
    "from keras.layers import Dropout\n",
    "from keras.layers import Activation\n",
    "from keras.regularizers import l2\n",
    "from keras.optimizers import SGD\n",
    "from keras.optimizers import RMSprop\n",
    "from keras import datasets\n",
    "\n",
    "from keras.callbacks import LearningRateScheduler\n",
    "from keras.callbacks import History\n",
    "\n",
    "from keras import losses\n",
    "from sklearn.utils import shuffle\n",
    "\n",
    "print(tf.VERSION)\n",
    "print(tf.keras.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# fix random seed for reproducibility\n",
    "np.random.seed(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1 -  Deciding on the network topology (not really considered optimization but is obviously very important)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use the MNIST dataset which consists of grayscale images of handwritten digits (0-9) whose dimension is 28x28 pixels. Each pixel is 8 bits so its value ranges from 0 to 255."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((60000, 28, 28), (60000,))"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#mnist = tf.keras.datasets.mnist\n",
    "mnist = keras.datasets.mnist\n",
    "(x_train, y_train),(x_test, y_test) = mnist.load_data()\n",
    "x_train.shape, y_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each label is a number between 0 and 9"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[5 0 4 ... 5 6 8]\n"
     ]
    }
   ],
   "source": [
    "print(y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at some 10 of the images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkAAAADuCAYAAADRE7iBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3XeUVEXax/GnHCWDqAQDYd5VJAgHUIxIEAUxkXTVXUEJKgZ0dsVFQFlYVl3EsAZwFQMgSDIgih4RA2EPoIASFURfB0FWARFQDKT7/gE+b9V1eranp7tvd9f3c86e/V1u9Z3HvTZTe+tWlQmCQAAAAHxySNQFAAAApBsdIAAA4B06QAAAwDt0gAAAgHfoAAEAAO/QAQIAAN6hAwQAALxDBwgAAHiHDhAAAPAOHSAAAOCdQ0vSuFq1akF+fn6KSkFRCgsLZevWrSbZ1+VeRmPp0qVbgyConuzrcj/Tj+9mbknFd5N7GY1472WJOkD5+fmyZMmSxKtCibVo0SIl1+VeRsMYsz4V1+V+ph/fzdySiu8m9zIa8d5LhsAAAIB36AABAADv0AECAADeoQMEAAC8QwcIAAB4hw4QAADwDh0gAADgHTpAAADAO3SAAACAd+gAAQAA75RoKwwg0yxdutQ5HjVqlObx48drvuaaa5x2t9xyi+aTTz45RdUBADIVT4AAAIB36AABAADv0AECAADeyal3gPbt2+cc79ixI67P2e+N/Pjjj5rXrl3rtBs9erTm22+/XfPkyZOdduXKldM8cOBA59zQoUPjqgmxLVu2TPN5553nnNu5c6dmY4zm5557zmk3Y8YMzdu2bUt2iYjQO++8o/mqq65yzs2dO1dz/fr101YTYrv77rud47/+9a+agyDQPGfOHKddmzZtUloXch9PgAAAgHfoAAEAAO9k7BDYl19+qXn37t3OuQULFmj+97//rXn79u1OuxdffLFUNdSuXds5tqdOT58+XXPlypWddk2bNtXMY9rk+OCDDzRfeumlmsPDnPawV5UqVTSXKVPGabd161bNCxcu1HzKKac47cKfyyXz5s1zjr/99lvNXbt2TXc5SbN48WLNLVq0iLASxDJu3DjNI0aMcM7l5eVptl9rsL/bQDLwBAgAAHiHDhAAAPBOxgyBffTRR85xu3btNMc7mysZ7Mev4dkJFStW1GzPLjn22GOddkcccYRmZprEz56B9+GHHzrnunfvrnnTpk1xXa9evXqaBwwY4Jy74oorNLds2VJz+J4PHjw4rp+VjcKzatatW6c524bA9u/fr/mLL77QbA+li7izihCd9evXa/7ll18irMRv77//vnM8YcIEzfYQ+apVq2Je48EHH9Qc/l04f/58zT169NB8+umnl7zYFOAJEAAA8A4dIAAA4B06QAAAwDsZ8w5Q3bp1neNq1appTsY7QPaYo/2OjojIe++9p9me9myPWSL1+vbtq3nSpEmlvp69U/wPP/zgnLOXJ7DfhVm5cmWpf262GD9+vHN81llnRVRJ6f3nP//RPGbMGM3h73CDBg3SVhNcb7/9tuZHH300Zjv7Hs2cOVNzzZo1U1OYZ6ZOnaq5oKDAObdlyxbN9vtybdu2ddrZy4jYuyKE2dewPzNlypT4C04hngABAADv0AECAADeyZghsCOPPNI5vv/++zW/9tprzrnmzZtrvvXWW2Nes1mzZprtx6/2dHYRd4pfcY9mkXz2MJX9uLu46cr249iLL77YOWc/jrWnZNr/zoi4w6D2EKhP06TtqePZ7tprry3yz+2lEJBe9ir9IiI9e/bUbG9aHPaXv/xFc/jVCMRn7969zrG9Ovp1112nedeuXU47+9WAIUOGaD777LOddvbSBZdffrnmWbNmxawpE1dl5wkQAADwDh0gAADgHTpAAADAOxnzDlBYly5dNNvbYoi4u6+vWLFC89NPP+20s98HCb/3Y2vcuLFmewotkm/ZsmXO8XnnnafZfi8gvPPzhRdeqHny5Mmaw9s53HPPPZrt90KqV6/utGvatGmRP+v111932tlbcpx88smS7ezvyzfffBNhJcm1ffv2Iv+8ffv2aa4EvwovsxBrC5vwFOurr746VSV5Y+LEic5xnz59imzXoUMH59ieIl+lSpWY17fbFffeT+3atTVfc801MdtFhSdAAADAO3SAAACAdzJ2CMxW3KO4ww8/POY5e0jsyiuv1HzIIfT70unTTz/VPHLkSOecvcq3PUx1zDHHOO3sx6eVKlXSHJ4GHz4uKXtHehGRBx54QHMyVqeO2htvvKH5p59+irCS0gkP3xUWFhbZ7rjjjktDNfiVvdrvM88845zLy8vTXLVqVc133XVX6gvzgP2/47333uucs4f5b775Zs133323066437U2+1WD4tjLyoRfQ8gE9AQAAIB36AABAADvZMUQWHGGDRum2V5VWMSdIWSvBB1+8x3JZa8SKuLOxgvPsrIfuT733HOaw6uGRjVcs2HDhkh+bqqsXbs25rmTTjopjZWUTngDxq+//lpz/fr1NdszRpEa9vBjt27d4vrMLbfcojk8yxfxGT58uHNsD3uVLVvWOXf++edrvu+++zSXL18+5vV//vlnzW+99ZZzbv369Zrt1fPt1aNFRDp37hzz+pmAJ0AAAMA7dIAAAIB36AABAADvZP07QPYKz0899ZRzzl65194B95xzznHa2e+b2FMEw6sRIz726skiv33vxzZjxgzN9k7ESL9TTz016hJ+s0v4m2++qdle3Tb8ToLNng5sT7dGatj3aOXKlTHbnXvuuZoLCgpSWlOuslc8f/zxx51z9u8r+50fEZFXXnklrut/9tlnmq+66irNS5YsifmZ3//+95oHDBgQ18/JFDwBAgAA3qEDBAAAvJP1Q2C2448/3jkeN26c5l69emm2p1uHj3ft2qU5vClfeHViFO22225zju1pkuGNDzNh2MuuryTncs22bdtK/Jnly5c7x/v379f8zjvvaN64caPTbvfu3Zqff/75Ij8v4k7TPf300zWHp/nu2bNHc3gJBSRXeDhl4MCBRbZr1aqVc2xvjlrcCv6Izf7ebNmyJWY7ewVmEZHNmzdrHjt2rGb7FQQRkdWrV2v+/vvvNYdfB7F3U+jevbvm4jYdz0Q8AQIAAN6hAwQAALyTU0NgYV27dtV8wgknaO7fv7/Tzl4letCgQZrt1S5FRO68807NbLLomjlzpuZly5Y55+zHp506dUpbTfGy6ws/6m3WrFm6y0kpe0gp/M/at29fzeHNFGMJD4HZQ4aHHXaY5goVKjjtGjZsqLl3796aTznlFKedPWRas2ZNzbVq1XLa2SuFN2jQIJ7SUQKJrPb8u9/9zjm27x8SU6ZMGc01atRwztnDXPn5+c65eGc027/X7FX6N23a5LSrVq2a5ksuuSSua2cingABAADv0AECAADeoQMEAAC8k9PvANmaNGmiedq0ac651157TXPPnj01P/HEE067devWaZ49e3aSK8xu9jsY9lRNEXes+oorrkhbTbbwDvXDhg0rsp29Wq2IyIgRI1JVUiTs1WPr1q3rnFuwYEGJr1enTh3n2N79uVGjRprPOOOMEl87bMyYMZrt9x1Efvu+CZLL3kE8Ly8vrs/Emh6PxNkrm4eXI7j44os1f/vtt845+x1Y+ztq/74TETnyyCM1X3nllZrD7wDZ57IZT4AAAIB36AABAADveDMEZgtvkNijRw/N1157rWZ7dVkRkXnz5mmeM2eO5vDqxnCVK1dOczpX07aHve6++27n3MiRIzXXrl1bc3iJhEqVKqWouujdcccdUZdQIvbK0mGXXXZZGivxg72cxaxZs+L6jL3MRf369ZNeE/6fvTK6SPErQ8fL/h03d+5czeFp9Lky5MwTIAAA4B06QAAAwDveDIGtWLFC84svvuicW7x4sebwsJfNntXSunXrJFaX29K5+rP92N4e5po6darTzp4J8fLLL6e+MKRUly5doi4h53To0EHzd999F7OdPRRjb3iK7GPP5i1uhXxmgQEAAGQpOkAAAMA7dIAAAIB3cuodoLVr1zrHjz32mGb7PY+vv/46rusdeqj7P489hfuQQ+g72uxdwO0s4q5Y+sgjjyT15z700EPO8d///nfNO3bs0Ny9e3en3XPPPZfUOoBcs3XrVs3Frf588803a87lZSN8cP7550ddQlrxWxwAAHiHDhAAAPBOVg6B2UNYkyZN0jxq1CinXWFhYYmvfeqpp2q+8847nXPpnM6dbYqbMmnfr1tvvdU517t3b81HHXWU5kWLFjntJkyYoHn58uWaN2zY4LSzN/js2LGj5ptuuqn4fwBkNXuj4jPPPDPCSrJXr169nGN7KHvfvn0xP3fWWWelrCakV7wrfucKngABAADv0AECAADeydghsG+++Ubz6tWrnXP9+vXTvGbNmhJfO7yJ3IABAzTbKwQz0ys59u7dq3n06NHOOXtV7sMPP1zzp59+Gte1w4/f27Vrp3n48OElqhPZa//+/VGXkJXsldNnz57tnLOHssuWLas5PJxcs2bNFFWHdPv888+jLiGt+A0PAAC8QwcIAAB4hw4QAADwTqTvAG3btk1z3759nXP22HSi45ItW7bU3L9/f83h1S7Lly+f0PXx/+ypx6eddppz7oMPPoj5OXuKvP3eV1i1atU02zsRJ3tlaWSnhQsXau7Zs2d0hWSZ7du3ay7u+3fsscdqfvDBB1NaE6LTqlUrzeEV/XMRT4AAAIB36AABAADvpHwI7P3333eOR44cqXnx4sWaN27cmND1K1SooDm8yrC9knPFihUTuj7iU6tWLc32xrMiIk8++aRme7PS4hQUFDjHN954o+Z69eolUiIAoBhNmjTRbP89G34NxT6uXr166gtLEZ4AAQAA79ABAgAA3qEDBAAAvJPyd4CmT59e7HEsjRo10nzJJZc45/Ly8jTffvvtmqtWrZpIiUiyY445xjkeNmxYkRkoiQsuuEDztGnTIqwkdzRo0EBzeFuZ+fPnp7scZJDBgwdr7tOnT8xzo0aN0mz/3s4GPAECAADeoQMEAAC8k/IhsBEjRhR7DADxsFd4ZrXn5Dj66KM1z507N8JKkGm6deumecqUKc652bNna7Zfaxg7dqzTLtOXn+EJEAAA8A4dIAAA4J1IN0MFAACZp0qVKprDsy7tXRYef/xxzeFZvpk+K4wnQAAAwDt0gAAAgHfoAAEAAO/wDhAAAIjJfh9IROSxxx4rMmcbngABAADv0AECAADeMUEQxN/YmC0isj515aAIdYMgqJ7si3IvI8P9zB3cy9yS9PvJvYxMXPeyRB0gAACAXMAQGAAA8A4dIAAA4J2c7wAZYwqNMSuNMcuMMUuirgelY4zpaIxZa4z5zBgzMOp6UDrGmDxjzEfGmJlR14LEGWOeNcZsNsasiroWlJ4xpsAYs8oYs9oY86eo60mVnO8AHXROEATNgiBoEXUhSJwxJk9ERovIBSLSSET+YIzJ7M1m8N8UiMgnUReBUhsnIh2jLgKlZ4xpLCLXichpItJURC42xtSLtqrU8KUDhNxwmoh8FgTB/wZBsFtEpohI54hrQoKMMbVE5CIReTrqWlA6QRDME5FtUdeBpGgoIouCIPgxCIK9IjJXRLpGXFNK+NABCkTkLWPMUmPM9VEXg1I5TkQ2WMcbD/4ZstPDIjJARPZHXQgAtUpEWhtjjjLGVBCRC0WkdsQ1pYQPW2G0DIJgkzGmhojMNsasOfj/VpB9TBF/xjoOWcgYc7GIbA6CYKkxpm3U9QA4IAiCT4wx94nIbBH5QUSWi8jeaKtKjZx/AhQEwaaD/71ZRKbLgWEUZKeN4v4/kVoisimiWlA6LUWkkzGmUA4MZbYzxkyMtiQAIiJBEDwTBMHJQRC0lgNDm+uirikVcroDZIypaIyp/GsWkQ5y4PEestNiEalnjPkfY0wZEblSRF6NuCYkIAiCQUEQ1AqCIF8O3Md3gyDoHnFZAETk4IiJGGPqiEg3EZkcbUWpketDYDVFZLoxRuTAP+ukIAjejLYkJCoIgr3GmH4iMktE8kTk2SAIVkdcFuA9Y8xkEWkrItWMMRtFZGgQBM9EWxVK4SVjzFEiskdEbg6C4LuoC0oFtsIAAADeyekhMAAAgKLQAQIAAN6hAwQAALxDBwgAAHiHDhAAAPAOHSAAAOCdEq0DVK1atSA/Pz9FpaAohYWFsnXr1qK2gCgV7mU0li5dujUIgurJvi73M/34buaWVHw3uZfRiPdelqgDlJ+fL0uWLEm8KpRYixYtUnJd7mU0jDHrU3Fd7mf68d3MLan4bnIvoxHvvWQIDAAAeIcOEAAA8A4dIAAA4B06QAAAwDt0gAAAgHfoAAEAAO/QAQIAAN6hAwQAALxDBwgAAHiHDhAAAPAOHSAAAOCdEu0FBqRLQUGBc/zoo49qbty4seaZM2c67erWrZvawgAASdWuXbuY5959992U/VyeAAEAAO/QAQIAAN6hAwQAALzj5TtA33//vXP8ww8/aH799dc1b9682WnXv39/zWXLlk1Rdf4qLCzUPGHCBOecMUbzxx9/rHnNmjVOO94Byhyffvqp5t27dzvn5s+fr/mmm27SbN/nRHXp0kXzlClTnHNlypQp9fV9t2fPHud4wYIFmgcNGlTknwO2P//5z87xwoULNV999dVpq4MnQAAAwDt0gAAAgHdyegjsiy++0Dxy5EjN9uM2EZGVK1fGdb2vv/5asz0tG8lRvXp1zW3atHHOzZgxI93lIA6rVq1yjsePH6/5hRde0Lx//36n3VdffaXZHvZKxhCY/e/KDTfc4Jx7+OGHNVepUqXUP8tHO3bscI7btm2r+eijj9Zs/30ZPgf/DBw4UPMTTzzhnDvssMM0n3vuuWmriSdAAADAO3SAAACAd7J+CMyeBWQ/3hYRmThxouaffvpJcxAETrs6deporly5smZ7tpGIyLRp0zTbM1caNGhQ0rJRhIoVK2pmNld2GDx4sHNsz6LMBPaQnIhI7969NZ999tnpLifn2cNeDIHBtmjRIs3hWaH2d/Hyyy9PW008AQIAAN6hAwQAALxDBwgAAHgnK94BCk+7vOOOOzRPnTpV886dO+O63oknnugcz5o1S7M9Nhl+t2fLli2at27dGtfPQvy2b9+uefny5RFWgni1b9/eOY71DlCNGjWc4z59+mi2p8gfckjs/09mryw8d+7cEtUJ4LfmzZvnHN9zzz2aJ0+erPnII49M6Pr2NezlZk444QSn3QMPPJDQ9UuLJ0AAAMA7dIAAAIB3smIIbPr06c7xU089VeJr2I/cZs+e7ZyrXbu25nXr1pX42kiOH3/8UfP69evj+szixYudY3vYkqn0qXfjjTc6x/ZGpDZ7pVeRxKZE20PcjRs3ds7ZK0sXV8+pp55a4p+LxNhLjyAzXX/99c6xvYGxvQxMoktG2ENq27Zt0/z000877Zo2bZrQ9UuLJ0AAAMA7dIAAAIB36AABAADvZMU7QPYWFMXJz893jk877TTN9913n2b7nZ8we2sNpNexxx6ruVevXs65oUOHFvmZ8J9XrVpVc79+/ZJYHYpy6KHuXyHFfbdKy16u4rvvvovrM+F6ypYtm9SaENvSpUud4zPPPDOiShBL+fLlnWNjjOaff/65xNdbtmyZc/zll18m7dqpwBMgAADgHTpAAADAO1kxBBaeMjdmzBjNHTp00BxeXTK8+mw8vvnmmxJ/Bsk3ZMgQ5zjWEBhy25QpUzTb33t7yYTiDB8+POk1+S487GkPO9uruX/++edpqwnxs/9uXbVqlXOuYcOGmuOdmr5r1y7N9qsm4XNnnHGG5ssuuyy+YlOMJ0AAAMA7dIAAAIB3smIIzJ4dJCIybNiwlP0se8NFZI4gCKIuASkyceJEzSNGjHDO2cMo9kbFxWnWrJnm8ArUKD17yEtEpFWrVppfe+21dJeDOGzYsEGzvZNCeDhz9OjRmqtXrx7XtW+77TbN4Rnbxx13nOZM/N3KEyAAAOAdOkAAAMA7dIAAAIB3suIdoEQ9+uijmu3peOH3SewVKsPTAm0tW7bUzKqm6WXfIzsjWoWFhc7xhAkTNL/99ttxXWP+/Pma4723VapUcY7t6bcXXnih5vBKt4APVq5c6Rx369ZN85YtWzTfeuutTrs2bdrEdf0HHnhA87hx42K2u/POO+O6XlR4AgQAALxDBwgAAHgnK4fA7FVgV69erTm86uvrr79e5OeLGwKzhaffjx07VnNeXl58xQI5xn683qlTJ+ecvflhKrVu3do5vv7669PycxG/b7/9NuoSctrevXudY3s5id69ezvn7N959u+7hQsXOu3uvfdezf3799e8bds2p90LL7xQ5LWvueYap13fvn1j/wNkAJ4AAQAA79ABAgAA3snYIbA9e/Zo/uijj5xzl156qeZNmzZprlChgtPOHsI666yzNL/55ptOO3uGmG3fvn3O8csvv6y5oKBAc5kyZYr8POCbRFbsTuQz4RWH33jjDc32LDBE59VXX426hJxmbxQsItKnTx/Nxc2mrFevnubFixc75+xj+/599dVXTjv796696fizzz7738rOKDwBAgAA3qEDBAAAvEMHCAAAeCdj3gEK7/Rsv6fTtWvXmJ+zd4Y/55xznHNnn322ZnsaX7t27Zx24VUzf7V582bneODAgZrr1KmjuUuXLk67smXLxqwXiYn3PZF58+Zp7tevX6rK8VqTJk00z5kzxzlnrwTdsWNHzeXKlUvoZz3zzDOa7ZXdkTnsv3fZDT61pk6dqrlXr17OOftd1KpVqzrnJk2apPmII47QbO/kLiIyd+5czfb7QMUtHbN161bNtWvXdtrZfz8cf/zxkml4AgQAALxDBwgAAHgn0iEwe6r70KFDnXMjR46M+bkLLrhA8y233KI5/NjP3vTNnhq7YsUKp509ZDVgwADN4aGxGTNmaP7jH/+ouX379k47+xr248aw5s2bxzwHV7ybob700kuaP/74Y82NGjVKTWGeq1u3rnN81113JfX69hA3Q2CZyX4dwBZ+rWH9+vWaw//eID5PPvmk5vBwk/3dC68EHcuoUaOcY3tF9fAq0bHs379fc/g1lEwc9rLxBAgAAHiHDhAAAPBO2ofA7NWVhwwZovn+++932lWqVEnzP/7xD+fcH/7wB832sFd4VUt7eOzDDz/UfOKJJzrt/vWvf2m2H+Ht3LnTabdgwQLNzz//vObwiqfhITGb/bj4iy++iNkOrhtuuEGz/Ri4OGPGjNH88MMPJ70mpN6sWbOiLgH/xaGHFv1rJDxz6JdffklHOTmtc+fOmrt16+acCw+JxcOewSXibi5uC6863bhx4yLb1apVq8Q1RIknQAAAwDt0gAAAgHfoAAEAAO+k/R0g+70M+72fihUrOu3s9zw6dOjgnFu0aJHmsWPHarZ3hBYR+emnnzTb0+zDK2jGGjutUqWKc2yvbGvnyZMnO+3s94PC/vnPf8Y8h9gaNmwYdQlesZeoCL+Hc+6552ouX758Un9ueDfpP/3pT0m9PpLPfi+lQYMGmtesWeO0s9/De/zxx1NfWA4qKCgo9TV27Nihedq0aTHPnXDCCZovv/zyUv/cTMQTIAAA4B06QAAAwDtpHwIbPnx4kX++d+9e59heCdpeDVZEZN26dXH9rL/97W+aBw0apDkvLy+uz8fLnpZf1DFKz17S4LHHHtP82WefxfzMI488UuTnRTJ/hdIozJ8/X/O9996r+a233nLaFRYWak5k6q2IuzmxPXTdv39/p92uXbuK/HyFChWc42QPxSEx559/vuZNmzY55x566KF0l4Mi2MOP9hIwIiI1a9bU/O6776atpqjwBAgAAHiHDhAAAPBO2ofAjj76aM2bN2/WHF4ldPny5TGvcdFFF2lu3bq15i5dujjt8vPzNSd72AvROemkkzR//vnnEVaSW+xhwvBGwDZ7eLpy5coJ/azZs2drXrp0qebiNrpt27at5ptuusk5F96EEdEL38syZcpEVAnsjWifeuopzYcc4j4DsTdDzbZVnRPBEyAAAOAdOkAAAMA7dIAAAIB30v4O0Lx58zS/8sormu3d2kVEatSoobl3797OuSOOOEIz48r+scepX3311Qgr8VMqV/G1v/ciIp06ddJsL2tQrly5lNWA5LBXFRZx/74P72SO1Grfvr1m+32gHj16OO3spWN8wBMgAADgHTpAAADAO2kfArOnzdqP38KP4oBYGjVqVGQWEfn444/TXU7OsDcWtlfbHj9+fKmvbW+sKOKu5NyqVSvN1113ndOuSZMmpf7ZSJ+pU6dqDg9Thr+rSJ+ePXtqHjJkiGZ7iNlHPAECAADeoQMEAAC8QwcIAAB4J+3vAAGlVbduXc3FbdmAkmnevLlme5fo008/3Wl31113abZ3dRdxt6Pp0KGD5s6dOzvt7C1xkDvatGmj+ZNPPnHOlS9fPt3l4KDBgwcXmX3HEyAAAOAdOkAAAMA7DIEB+I2yZctq7tu3r3MufAz8asqUKVGXAMSNJ0AAAMA7dIAAAIB36AABAADv0AECAADeoQMEAAC8QwcIAAB4hw4QAADwDh0gAADgHTpAAADAOyYIgvgbG7NFRNanrhwUoW4QBNWTfVHuZWS4n7mDe5lbkn4/uZeRietelqgDBAAAkAsYAgMAAN6hAwQAALyT0x0gY0xtY8x7xphPjDGrjTEFUdeExBljnjXGbDbGrIq6FpSOMaacMeYDY8zyg9/Nv0VdExLHdzP3GGPyjDEfGWNmRl1LquR0B0hE9opI/yAIGorIGSJyszGmUcQ1IXHjRKRj1EUgKX4RkXZBEDQVkWYi0tEYc0bENSFx44TvZq4pEJFPoi4ilXK6AxQEwX+CIPjwYP5eDtzM46KtCokKgmCeiGyLug6UXnDADwcPDzv4H2ZkZCm+m7nFGFNLRC4SkaejriWVcroDZDPG5ItIcxF5P9pKAIjoI/ZlIrJZRGYHQcB3E8gMD4vIABHZH3UhqeRFB8gYU0lEXhKRPwVBsDPqegCIBEGwLwiCZiJSS0ROM8Y0jromwHfGmItFZHMQBEujriXVcr4DZIw5TA50fp4PguDlqOsB4AqCYLsK+7DPAAAA1UlEQVSIzBHeIQEyQUsR6WSMKRSRKSLSzhgzMdqSUiOnO0DGGCMiz4jIJ0EQPBR1PQAOMMZUN8ZUPZjLi8h5IrIm2qoABEEwKAiCWkEQ5IvIlSLybhAE3SMuKyVyugMkB3qyPeRAD3bZwf9cGHVRSIwxZrKILBSR+saYjcaYPlHXhIQdIyLvGWNWiMhiOfAOUM5Ot811fDeRjdgKAwAAeCfXnwABAAD8Bh0gAADgHTpAAADAO3SAAACAd+gAAQAA79ABAgAA3qEDBAAAvEMHCAAAeOf/ADaW/GIW4/wlAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x720 with 10 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(10,10))\n",
    "for i in range(10):\n",
    "    plt.subplot(5,5,i+1)\n",
    "    plt.xticks([])\n",
    "    plt.yticks([])\n",
    "    plt.grid(False)\n",
    "    plt.imshow(x_train[i], cmap=plt.cm.binary)\n",
    "    plt.xlabel(y_train[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 11, 198, 231,  41,   0],\n",
       "       [ 82, 252, 204,   0,   0],\n",
       "       [253, 253, 141,   0,   0],\n",
       "       [252, 220,  36,   0,   0],\n",
       "       [252,  96,   0,   0,   0]], dtype=uint8)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train[45].shape\n",
    "x_train[45, 15:20, 15:20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "We have 60000 train samples\n",
      "We have 10000 test samples\n"
     ]
    }
   ],
   "source": [
    "print(f'We have {x_train.shape[0]} train samples')\n",
    "print(f'We have {x_test.shape[0]} test samples')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Preprocessing the data\n",
    "\n",
    "To run our NN we need to pre-process the data\n",
    "\n",
    "* First we need to make the 2D image arrays into 1D (flatten them). We can either perform this by using array reshaping with `numpy.reshape()` or the `keras`' method for this: a layer called `tf.keras.layers.Flatten` which transforms the format of the images from a 2d-array (of 28 by 28 pixels), to a 1D-array of 28 * 28 = 784 pixels. \n",
    "\n",
    "* Then we need to normalize the pixel values (give them values between 0 and 1) using the following transformation:\n",
    "\n",
    "\\begin{align}\n",
    "x := \\dfrac{x - x_{min}}{x_{max} - x_{min}} \n",
    "\\textrm{}\n",
    "\\end{align}\n",
    "\n",
    "In our case $x_{min} = 0$ and $x_{max} = 255$ so the formula becomes simply $x := {x}/255$ "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "code_folding": []
   },
   "outputs": [],
   "source": [
    "# normalize the data\n",
    "x_train, x_test = x_train / 255.0, x_test / 255.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# reshape the data into 1D vectors\n",
    "x_train = x_train.reshape(60000, 784)\n",
    "x_test = x_test.reshape(10000, 784)\n",
    "\n",
    "num_classes = 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "784"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train.shape[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's prepare our class vector (y) to a binary class matrix, e.g. for use with categorical_crossentropy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert class vectors to binary class matrices\n",
    "y_train = keras.utils.to_categorical(y_train, num_classes)\n",
    "y_test = keras.utils.to_categorical(y_test, num_classes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 0., 0., 0., 0., 1., 0., 0., 0., 0.], dtype=float32)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are ready to build the model!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2 - Adjusting the `learning rate`\n",
    "\n",
    "One of the most common optimization algorithm is Stochastic Gradient Descent (SGD). The hyperparameters that can be optimized in SGD are `learning rate`, `momentum`, `decay` and `nesterov`.\n",
    "\n",
    "`Learning rate` controls the weight at the end of each batch, and `momentum` controls how much to let the previous update influence the current weight update. `Decay` indicates the learning rate decay over each update, and `nesterov` takes the value True or False depending on if we want to apply Nesterov momentum. Typical values for those hyperparameters are lr=0.01, decay=1e-6, momentum=0.9, and nesterov=True. \n",
    "\n",
    "The learning rate hyperparameter goes into the `optimizer` function which we will see below. Keras has a default learning rate scheduler in the `SGD` optimizer that decreases the learning rate during the  stochastic gradient descent optimization algorithm. The learning rate is decreased according to this formula:\n",
    "\n",
    "\\begin{align}\n",
    "lr = lr * 1./(1. + decay * epoch)\n",
    "\\textrm{}\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![learning rates](learningrates.jpeg)\n",
    "\n",
    "<pre>source: http://cs231n.github.io/neural-networks-3</pre>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's implement a learning rate adaptation schedule in `Keras`. We'll start with SGD and a learning rate value of 0.1. We will then train the model for 60\n",
    "epochs and set the decay argument to 0.0016 (0.1/60). We also include a momentum value of 0.8 since that seems to work well when using an adaptive learning rate. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "epochs=60\n",
    "learning_rate = 0.1\n",
    "decay_rate = learning_rate / epochs\n",
    "momentum = 0.8\n",
    "\n",
    "sgd = SGD(lr=learning_rate, momentum=momentum, decay=decay_rate, nesterov=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# build the model\n",
    "input_dim = x_train.shape[1]\n",
    "\n",
    "lr_model = Sequential()\n",
    "lr_model.add(Dense(64, activation=tf.nn.relu, kernel_initializer='uniform', \n",
    "                input_dim = input_dim)) \n",
    "lr_model.add(Dropout(0.1))\n",
    "lr_model.add(Dense(64, kernel_initializer='uniform', activation=tf.nn.relu))\n",
    "lr_model.add(Dense(num_classes, kernel_initializer='uniform', activation=tf.nn.softmax))\n",
    "\n",
    "# compile the model\n",
    "lr_model.compile(loss='categorical_crossentropy',\n",
    "              optimizer=sgd,\n",
    "              metrics=['acc'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 60000 samples, validate on 10000 samples\n",
      "Epoch 1/60\n",
      "60000/60000 [==============================] - 9s 145us/step - loss: 0.3158 - acc: 0.9043 - val_loss: 0.1467 - val_acc: 0.9550\n",
      "Epoch 2/60\n",
      "60000/60000 [==============================] - 8s 136us/step - loss: 0.1478 - acc: 0.9555 - val_loss: 0.1194 - val_acc: 0.9617\n",
      "Epoch 3/60\n",
      "60000/60000 [==============================] - 8s 137us/step - loss: 0.1248 - acc: 0.9620 - val_loss: 0.1122 - val_acc: 0.9646\n",
      "Epoch 4/60\n",
      "60000/60000 [==============================] - 8s 136us/step - loss: 0.1167 - acc: 0.9638 - val_loss: 0.1075 - val_acc: 0.9681\n",
      "Epoch 5/60\n",
      "60000/60000 [==============================] - 8s 137us/step - loss: 0.1103 - acc: 0.9666 - val_loss: 0.1039 - val_acc: 0.9691\n",
      "Epoch 6/60\n",
      "60000/60000 [==============================] - 8s 137us/step - loss: 0.1051 - acc: 0.9677 - val_loss: 0.1015 - val_acc: 0.9694\n",
      "Epoch 7/60\n",
      "60000/60000 [==============================] - 8s 136us/step - loss: 0.1003 - acc: 0.9691 - val_loss: 0.1002 - val_acc: 0.9694\n",
      "Epoch 8/60\n",
      "60000/60000 [==============================] - 9s 144us/step - loss: 0.0961 - acc: 0.9707 - val_loss: 0.0998 - val_acc: 0.9694\n",
      "Epoch 9/60\n",
      "60000/60000 [==============================] - 9s 154us/step - loss: 0.0951 - acc: 0.9707 - val_loss: 0.0989 - val_acc: 0.9699\n",
      "Epoch 10/60\n",
      "60000/60000 [==============================] - 9s 150us/step - loss: 0.0919 - acc: 0.9721 - val_loss: 0.0978 - val_acc: 0.9696\n",
      "Epoch 11/60\n",
      "60000/60000 [==============================] - 8s 141us/step - loss: 0.0930 - acc: 0.9720 - val_loss: 0.0964 - val_acc: 0.9702\n",
      "Epoch 12/60\n",
      "60000/60000 [==============================] - 8s 141us/step - loss: 0.0899 - acc: 0.9728 - val_loss: 0.0965 - val_acc: 0.9703\n",
      "Epoch 13/60\n",
      "60000/60000 [==============================] - 8s 141us/step - loss: 0.0883 - acc: 0.9732 - val_loss: 0.0951 - val_acc: 0.9713\n",
      "Epoch 14/60\n",
      "60000/60000 [==============================] - 8s 141us/step - loss: 0.0871 - acc: 0.9733 - val_loss: 0.0958 - val_acc: 0.9705\n",
      "Epoch 15/60\n",
      "60000/60000 [==============================] - 8s 141us/step - loss: 0.0888 - acc: 0.9731 - val_loss: 0.0952 - val_acc: 0.9709\n",
      "Epoch 16/60\n",
      "60000/60000 [==============================] - 9s 145us/step - loss: 0.0857 - acc: 0.9743 - val_loss: 0.0950 - val_acc: 0.9713\n",
      "Epoch 17/60\n",
      "60000/60000 [==============================] - 9s 157us/step - loss: 0.0843 - acc: 0.9742 - val_loss: 0.0957 - val_acc: 0.9709\n",
      "Epoch 18/60\n",
      "60000/60000 [==============================] - 8s 142us/step - loss: 0.0842 - acc: 0.9749 - val_loss: 0.0942 - val_acc: 0.9719\n",
      "Epoch 19/60\n",
      "60000/60000 [==============================] - 9s 142us/step - loss: 0.0839 - acc: 0.9750 - val_loss: 0.0936 - val_acc: 0.9723\n",
      "Epoch 20/60\n",
      "60000/60000 [==============================] - 9s 142us/step - loss: 0.0824 - acc: 0.9748 - val_loss: 0.0942 - val_acc: 0.9723\n",
      "Epoch 21/60\n",
      "60000/60000 [==============================] - 9s 143us/step - loss: 0.0824 - acc: 0.9749 - val_loss: 0.0940 - val_acc: 0.9725\n",
      "Epoch 22/60\n",
      "60000/60000 [==============================] - 9s 142us/step - loss: 0.0829 - acc: 0.9752 - val_loss: 0.0938 - val_acc: 0.9718\n",
      "Epoch 23/60\n",
      "60000/60000 [==============================] - 9s 143us/step - loss: 0.0795 - acc: 0.9763 - val_loss: 0.0939 - val_acc: 0.9718\n",
      "Epoch 24/60\n",
      "60000/60000 [==============================] - 9s 149us/step - loss: 0.0796 - acc: 0.9763 - val_loss: 0.0936 - val_acc: 0.9722\n",
      "Epoch 25/60\n",
      "60000/60000 [==============================] - 8s 140us/step - loss: 0.0783 - acc: 0.9759 - val_loss: 0.0935 - val_acc: 0.9724\n",
      "Epoch 26/60\n",
      "60000/60000 [==============================] - 8s 140us/step - loss: 0.0805 - acc: 0.9755 - val_loss: 0.0937 - val_acc: 0.9721\n",
      "Epoch 27/60\n",
      "60000/60000 [==============================] - 8s 140us/step - loss: 0.0795 - acc: 0.9759 - val_loss: 0.0930 - val_acc: 0.9721\n",
      "Epoch 28/60\n",
      "60000/60000 [==============================] - 9s 148us/step - loss: 0.0786 - acc: 0.9765 - val_loss: 0.0931 - val_acc: 0.9721\n",
      "Epoch 29/60\n",
      "60000/60000 [==============================] - 9s 148us/step - loss: 0.0780 - acc: 0.9764 - val_loss: 0.0926 - val_acc: 0.9726\n",
      "Epoch 30/60\n",
      "60000/60000 [==============================] - 9s 144us/step - loss: 0.0759 - acc: 0.9768 - val_loss: 0.0925 - val_acc: 0.9726\n",
      "Epoch 31/60\n",
      "60000/60000 [==============================] - 9s 147us/step - loss: 0.0780 - acc: 0.9768 - val_loss: 0.0931 - val_acc: 0.9727\n",
      "Epoch 32/60\n",
      "60000/60000 [==============================] - 9s 155us/step - loss: 0.0768 - acc: 0.9765 - val_loss: 0.0925 - val_acc: 0.9726\n",
      "Epoch 33/60\n",
      "60000/60000 [==============================] - 9s 144us/step - loss: 0.0762 - acc: 0.9770 - val_loss: 0.0927 - val_acc: 0.9723\n",
      "Epoch 34/60\n",
      "60000/60000 [==============================] - 9s 149us/step - loss: 0.0765 - acc: 0.9770 - val_loss: 0.0928 - val_acc: 0.9723\n",
      "Epoch 35/60\n",
      "60000/60000 [==============================] - 8s 140us/step - loss: 0.0751 - acc: 0.9778 - val_loss: 0.0928 - val_acc: 0.9721\n",
      "Epoch 36/60\n",
      "60000/60000 [==============================] - 8s 138us/step - loss: 0.0756 - acc: 0.9772 - val_loss: 0.0919 - val_acc: 0.9721\n",
      "Epoch 37/60\n",
      "60000/60000 [==============================] - 8s 140us/step - loss: 0.0760 - acc: 0.9769 - val_loss: 0.0923 - val_acc: 0.9723\n",
      "Epoch 38/60\n",
      "60000/60000 [==============================] - 8s 138us/step - loss: 0.0751 - acc: 0.9772 - val_loss: 0.0921 - val_acc: 0.9726\n",
      "Epoch 39/60\n",
      "60000/60000 [==============================] - 9s 148us/step - loss: 0.0756 - acc: 0.9774 - val_loss: 0.0924 - val_acc: 0.9728\n",
      "Epoch 40/60\n",
      "60000/60000 [==============================] - 8s 141us/step - loss: 0.0750 - acc: 0.9774 - val_loss: 0.0924 - val_acc: 0.9728\n",
      "Epoch 41/60\n",
      "60000/60000 [==============================] - 9s 142us/step - loss: 0.0760 - acc: 0.9774 - val_loss: 0.0926 - val_acc: 0.9724\n",
      "Epoch 42/60\n",
      "60000/60000 [==============================] - 8s 142us/step - loss: 0.0719 - acc: 0.9783 - val_loss: 0.0920 - val_acc: 0.9730\n",
      "Epoch 43/60\n",
      "60000/60000 [==============================] - 9s 143us/step - loss: 0.0730 - acc: 0.9779 - val_loss: 0.0919 - val_acc: 0.9726\n",
      "Epoch 44/60\n",
      "60000/60000 [==============================] - 8s 140us/step - loss: 0.0722 - acc: 0.9785 - val_loss: 0.0920 - val_acc: 0.9728\n",
      "Epoch 45/60\n",
      "60000/60000 [==============================] - 9s 142us/step - loss: 0.0746 - acc: 0.9774 - val_loss: 0.0923 - val_acc: 0.9730\n",
      "Epoch 46/60\n",
      "60000/60000 [==============================] - 9s 148us/step - loss: 0.0736 - acc: 0.9778 - val_loss: 0.0920 - val_acc: 0.9729\n",
      "Epoch 47/60\n",
      "60000/60000 [==============================] - 9s 156us/step - loss: 0.0739 - acc: 0.9777 - val_loss: 0.0920 - val_acc: 0.9725\n",
      "Epoch 48/60\n",
      "60000/60000 [==============================] - 9s 151us/step - loss: 0.0720 - acc: 0.9783 - val_loss: 0.0917 - val_acc: 0.9731\n",
      "Epoch 49/60\n",
      "60000/60000 [==============================] - 9s 146us/step - loss: 0.0735 - acc: 0.9780 - val_loss: 0.0917 - val_acc: 0.9729\n",
      "Epoch 50/60\n",
      "60000/60000 [==============================] - 9s 152us/step - loss: 0.0729 - acc: 0.9780 - val_loss: 0.0923 - val_acc: 0.9723\n",
      "Epoch 51/60\n",
      "60000/60000 [==============================] - 9s 151us/step - loss: 0.0716 - acc: 0.9777 - val_loss: 0.0919 - val_acc: 0.9727\n",
      "Epoch 52/60\n",
      "60000/60000 [==============================] - 9s 145us/step - loss: 0.0716 - acc: 0.9784 - val_loss: 0.0915 - val_acc: 0.9726\n",
      "Epoch 53/60\n",
      "60000/60000 [==============================] - 9s 149us/step - loss: 0.0715 - acc: 0.9782 - val_loss: 0.0912 - val_acc: 0.9722\n",
      "Epoch 54/60\n",
      "60000/60000 [==============================] - 9s 143us/step - loss: 0.0704 - acc: 0.9786 - val_loss: 0.0911 - val_acc: 0.9720\n",
      "Epoch 55/60\n",
      "60000/60000 [==============================] - 9s 142us/step - loss: 0.0721 - acc: 0.9782 - val_loss: 0.0917 - val_acc: 0.9727\n",
      "Epoch 56/60\n",
      "60000/60000 [==============================] - 9s 143us/step - loss: 0.0717 - acc: 0.9784 - val_loss: 0.0918 - val_acc: 0.9725\n",
      "Epoch 57/60\n",
      "60000/60000 [==============================] - 9s 151us/step - loss: 0.0717 - acc: 0.9783 - val_loss: 0.0918 - val_acc: 0.9726\n",
      "Epoch 58/60\n",
      "60000/60000 [==============================] - 9s 144us/step - loss: 0.0708 - acc: 0.9783 - val_loss: 0.0916 - val_acc: 0.9725\n",
      "Epoch 59/60\n",
      "60000/60000 [==============================] - 8s 137us/step - loss: 0.0703 - acc: 0.9782 - val_loss: 0.0916 - val_acc: 0.9731\n",
      "Epoch 60/60\n",
      "60000/60000 [==============================] - 8s 137us/step - loss: 0.0703 - acc: 0.9785 - val_loss: 0.0918 - val_acc: 0.9727\n",
      "CPU times: user 15min 16s, sys: 3min 41s, total: 18min 57s\n",
      "Wall time: 8min 37s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# Fit the model\n",
    "batch_size = int(input_dim/100)\n",
    "\n",
    "lr_model_history = lr_model.fit(x_train, y_train,\n",
    "                    batch_size=batch_size,\n",
    "                    epochs=epochs,\n",
    "                    verbose=1,\n",
    "                    validation_data=(x_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAooAAAGJCAYAAAANCBK/AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3XmclXX5//HXBYwsM8MOiiLgCm64gIKMGooh2tfdzFL7ppWallvrr1Va7VuZaZlLmpl7Vmra4oZoihoopomIGJugssjOsM31++M6t3M4zHJm5qzD+/l43I975l4/50T45rOauyMiIiIikqlDsQsgIiIiIqVJQVFEREREGqSgKCIiIiINUlAUERERkQYpKIqIiIhIgxQURURERKRBCooiIiIi0iAFRRERERFpkIKiiIiIiDRIQVFEREREGtSp2AVoL/r27etDhgwpdjFEREREmjVt2rQl7t6vuesUFHNkyJAhTJ06tdjFEBEREWmWmc3N5jo1PYuIiIhIgxQURURERKRBCooiIiIi0iD1URQREZFtzsaNG1mwYAG1tbXFLkpedenShYEDB1JRUdGq+xUURUREZJuzYMECqqurGTJkCGZW7OLkhbuzdOlSFixYwC677NKqZ6jpWURERLY5tbW19OnTp92GRAAzo0+fPm2qNVVQFBERkW1Sew6JibZ+RgVFERERkQJbvnw51113XYvvO+6441i+fHkeStQwBUURERGRAmssKG7evLnJ+/7617/Ss2fPfBVrKxrMIiIiIlJgX/va15g9ezYHHHAAFRUVVFVVMWDAAKZPn85rr73GSSedxPz586mtreWSSy7hvPPOA+pXglu9ejXHHnsshx12GM8++yw77bQTDzzwAF27ds1pORUURUREZNt26aUwfXpun3nAAXD11Y2evvLKK3n11VeZPn06Tz75JB/5yEd49dVXPxidfMstt9C7d2/WrVvHwQcfzKmnnkqfPn22eMasWbO46667uOmmmzj99NP54x//yFlnnZXTj6Gm53Ixdy489BA0UyUtIiIi5eeQQw7ZYgqba665hv3335/Ro0czf/58Zs2atdU9u+yyCwcccAAAI0aMYM6cOTkvl2oUy8Wf/wyXXQbLlkGvXsUujYiISPvRRM1foVRWVn7w85NPPsljjz3GlClT6NatG2PHjm1wipvOnTt/8HPHjh1Zt25dzsulGsVyUVUV+9Wri1sOERERabPq6mpWrVrV4LkVK1bQq1cvunXrxuuvv85zzz1X4NLVU41iuUiCYiN/qERERKR89OnTh5qaGvbdd1+6du3K9ttv/8G5CRMmcP311zN8+HCGDh3K6NGji1ZOBcVyUV0de9UoioiItAt33nlng8c7d+7M3/72twbPJf0Q+/bty6uvvvrB8S996Us5Lx+o6bl8qEZRRERECkxBsVyoRlFEREQKrGSCopkNNLNbzGyhma03szlmdrWZZT3E18yeNDNvYuvSwD1NXV+83qOZNJhFRERECqwk+iia2W7As0B/4AHgdeAQ4BJggpnVuPvSFjxyYiPHNzVyfC5wawPHF7TgnfmlpmcREREpsJIIisB1REi82N2vTQ6a2VXAZcAPgAuyfZi7X9HC989pxT2FpaZnERERKbCiNz2b2a7AeGAO8KuM098B1gBnm1kl27JkIk7VKIqIiEiBlEKN4lGp/SPuXpd+wt1XmdkzRJAcDTyezQPN7GPALsAGYAbwhLuvb+KWnmZ2LrADsAKY5u6l0z8RoEOHCIuqURQREdnmVFVVsboIGaAUguLQ1P6NRs7PIoLinmQZFIG7M35/z8wucvf7Grl+f+Dm9ANm9jJwtru/kuU786+qSkFRRERECqboTc9Aj9R+RSPnk+M9s3jWA8DxwECgKzAM+FHq3nvM7NgG7rkKqAH6AdXAwcB9RHh8wsx2auxlZnaemU01s6mLFy/OonhtVF2tpmcREZF24Ktf/SrXXXfdB79fccUVTJw4kXHjxnHQQQex33778cADDxSxhKEUahSbY6m9N3ehu/8849BM4OtmthC4Fvgh8LeMe76Ycc9U4KNmdh9wKvAlYkBNQ++7EbgRYOTIkc2Wr81UoygiIpJzl14K06fn9pkHHABXX934+TPOOINLL72UCy+8EIB7772Xv//971x22WV0796dJUuWMHr0aE444QTMrPEH5Vkp1CgmNYY9GjnfPeO61vgNMTXOAWZWneU916f2R7ThvblVVaUaRRERkXbgwAMP5L333mPhwoW8/PLL9OrViwEDBvD1r3+d4cOHc/TRR/P222/z7rvvFrWcpVCjODO137OR83uk9o31YWyWu9ea2SqgF1AJZJO2krbk0hltXV0NhWjiFhER2YY0VfOXT6eddhr33Xcf77zzDmeccQZ33HEHixcvZtq0aVRUVDBkyBBqa2uLU7iUUqhRnJTajzezLcqTqv2rAdYBrR6FbGZDiZC4CliS5W2jU/u3WvvenFPTs4iISLtxxhlncPfdd3Pfffdx2mmnsWLFCvr3709FRQWTJk1i7ty5xS5i8YOiu88GHgGGABdlnJ5I1Ojd5u5rkoNmNszMhqVfaGa7NjTwxMz6Ar9N/Xq3u29KO3dQQ/MzmtlwYpJvgNtb/KHyRYNZRERE2o199tmHVatWsdNOOzFgwADOPPNMpk6dysiRI7njjjsYNmxY8w/Js1Joega4kFjC7xozG0fMfTgKOJJocv5GxvUzUvv03p1HAL8xs8nAbGAZMAg4juj/OBX4SsZzLgZOMbMngPnAemKk9ASgI3ATcFcOPl9uqEZRRESkXXnllfpZ+Pr27cuUKVMavK4YcyhCiQRFd59tZiOB7xIh7ThgEXANMNHdl2XxmGlE7d8I4ABiEMwq4BXgXuAGd9+Qcc/9qeuGExN/dwGWEiOjb3L3B9v40XIrGcziDkUcASUiIiLbhpIIigDuPh84J8trt0pJqYmxP9XCd95PhMXyUF0NdXVQWwtduxa7NCIiItLOFb2PorRAVVXs1U9RRERECkBBsZxUp6aAVD9FERGRNnPP/1oZxdbWz6igWE6SGkUFRRERkTbp0qULS5cubddh0d1ZunQpXbp0afUzSqaPomQhqVFU07OIiEibDBw4kAULFrC4nS9k0aVLFwYOHNjq+xUUy4lqFEVERHKioqKCXXbZpdjFKHlqei4nGswiIiIiBaSgWE40mEVEREQKSEGxnKjpWURERApIQbGcaDCLiIiIFJCCYjnp3Bk6dlSNooiIiBSEgmI5Matf71lEREQkzxQUy011tWoURUREpCAUFMtNVZWCooiIiBSEgmK5qa5W07OIiIgUhIJiuVGNooiIiBSIgmK5UY2iiIiIFIiCYrlRjaKIiIgUiIJiudH0OCIiIlIgCorlRtPjiIiISIEoKJabqipYuxY2by52SURERKSdU1AsN8l6z2vWFLccIiIi0u4pKJabqqrYq/lZRERE8kxBsdwkQVEDWkRERCTPFBTLTdL0rBpFERERyTMFxXKjpmcREREpEAXFcpPUKKrpWURERPJMQbHcqEZRRERECkRBsdyoRlFEREQKREGx3KhGUURERApEQbHcKCiKiIhIgSgolptOnaBLFzU9i4iISN4pKJajqirVKIqIiEjeKSiWo+pq1SiKiIhI3ikoliPVKIqIiEgBKCiWo6oq1SiKiIhI3pVMUDSzgWZ2i5ktNLP1ZjbHzK42s14teMaTZuZNbF0auW9vM7vXzN4zs1ozm2lmE82sa+4+YQ5VV6tGUURERPKuU7ELAGBmuwHPAv2BB4DXgUOAS4AJZlbj7ktb8MiJjRzf1MC7RwFPABXAfcB84Cjg28A4Mxvn7utb8O78q6qCt98udilERESknSuJoAhcR4TEi9392uSgmV0FXAb8ALgg24e5+xXZXGdmHYHfAt2AE939wdTxDsC9wKmp91+Z7bsLQoNZREREpACK3vRsZrsC44E5wK8yTn8HWAOcbWaVeXj9h4C9gKeSkAjg7nXAV1K/XmBmlod3t54Gs4iIiEgBFD0oEs28AI+kAtoH3H0V8AxR4zc62wea2cfM7GtmdrmZHWtmnZt5998zT7j7W8AbwGBg12zfXRCqURQREZECKIWgODS1f6OR87NS+z1b8My7gR8BPwP+Cswzs9MK9O78q6qCjRthw4Zil0RERETasVIIij1S+xWNnE+O98ziWQ8AxwMDga7AMCIw9gTuMbNjc/luMzvPzKaa2dTFixdnUbwc0XrPIiIiUgClEBSbk/QP9OYudPefu/tD7v62u9e6+0x3/zrwReKz/jCX73b3G919pLuP7NevXwsf3QbV1bFX87OIiIjkUSkExaTWrkcj57tnXNcavyGmxjnAzKoL/O7cU42iiIiIFEApBMWZqX1j/QD3SO0b60fYLHevBZLqt/TR03l/d16oRlFEREQKoBSC4qTUfnxq/sIPpGr/aoB1wHOtfYGZDQV6EWFxSdqpJ1L7CQ3csysRIOcCb7X23XmhGkUREREpgKIHRXefDTwCDAEuyjg9kagBvM3d1yQHzWyYmQ1Lv9DMdjWznTKfb2Z9iUm1Ae529/TVWSYDM4AjzOyEtHs6AD9O/Xq9uzfbP7KgFBRFRESkAEplZZYLiSX8rjGzcUR4GwUcSTT7fiPj+hmpffpE2EcAvzGzycBsYBkwCDiO6IM4lfpJtAFw981mdg5Rs3ifmd0HzAPGASOJORx/nqPPmDtqehYREZECKImg6O6zzWwk8F2iGfg4YBFwDTDR3Zdl8ZhpwO3ACOAAYiDKKuAVYjm+G9x9q4kH3f15MzuYqL0cD1QTzc3fBa4suXWeQTWKIiIiUhAlERQB3H0+cE6W1261pJ67vwJ8qpXvfg34aGvuLQrVKIqIiEgBFL2PorRCt25gphpFERERySsFxXJkFs3PCooiIiKSRwqK5aqqSk3PIiIiklcKiuVKNYoiIiKSZwqK5aq6WjWKIiIiklcKiuVKNYoiIiKSZwqK5Uo1iiIiIpJnCorlSjWKIiIikmcKiuVKQVFERETyTEGxXKnpWURERPJMQbFcJTWK7sUuiYiIiLRTCorlqro6QuLatcUuiYiIiLRTCorlqqoq9uqnKCIiInmioFiuqqtjr6AoIiIieaKgWK6SGkUNaBEREZE8UVAsV2p6FhERkTxTUCxXSdOzahRFREQkTxQUy5VqFEVERCTPFBTLlQaziIiISJ4pKJYrDWYRERGRPFNQLFdqehYREZE8U1AsV507Q0WFahRFREQkbxQUy1my3rOIiIhIHigolrPqatUoioiISN4oKJYz1SiKiIhIHikolrPqagVFERERyRsFxXJWVaWmZxEREckbBcVypqZnERERySMFxXKmwSwiIiKSRwqK5Uw1iiIiIpJHCorlTINZREREJI8UFMtZVRWsWwebNhW7JCIiItIOKSiWs2S95zVrilsOERERaZcUFMtZdXXsNaBFRERE8kBBsZwlNYrqpygiIiJ5UDJB0cwGmtktZrbQzNab2Rwzu9rMerXhmUeY2WYzczP7fgPnh6TONbbd3bZPlWdJjaKCooiIiORBp2IXAMDMdgOeBfoDDwCvA4cAlwATzKzG3Ze28JnVwO+AtUBVM5e/DNzfwPFXW/LOgktqFNX0LCIiInlQEkERuI4IiRe7+7XJQTO7CrgM+AFwQQuf+QugB/Cj1P1Nme7uV7Tw+cWnGkURERHJo6I3PZvZrsB4YA7wq4zT3wHWAGebWWULnnkicA5wMbAwNyUtQapRFBERkTwqelAEjkrtH3H3uvQT7r4KeAboBozO5mFm1h+4Cbjf3W/Psgw7mtn5Zvb11H54lvcVlwaziIiISB6VQtPz0NT+jUbOzyJqHPcEHs/ieTcSAbglTdUfTm0fMLMngf9193kteE5haXocERERyaNSqFHskdqvaOR8crxncw8ys3OBE4EL3f3dLN69FvgeMALoldo+BEwCxgKPN9XkbWbnmdlUM5u6ePHiLF6XY5WpoqlGUURERPKgFIJicyy19yYvMhsCXA38wd3vzebB7v6eu3/b3V909+Wp7SmiBvN5YHfgM03cf6O7j3T3kf369cvmlbnVsSN066agKCIiInlRCkExqTHs0cj57hnXNeYWYB1wYVsL5O6bgN+kfj2irc/Lq6oqNT2LiIhIXpRCUJyZ2u/ZyPk9UvvG+jAmDiKm2FmcPmk28NvU+W+kjjU0X2JDkrbkrEdbF0VVlWoURUREJC9KYTDLpNR+vJl1SB/5nJo0u4aoKXyumefcRoyOzrQHUSs4HZgGvJRluZJR1m9leX1xVFerRlFERETyouhB0d1nm9kjRL/Ai4Br005PJGr0bnD3NclBMxuWuvf1tOdc3NDzzexTRFB82N2/mXFuFPCSu2/IOH4UMdE3QLZT7BSHahRFREQkT4oeFFMuJJbwu8bMxgEzgFHAkUST8zcyrp+R2htt82Ngn9RUOAtSx4ZTP7fjt9z92Ta+I7+qq2HZsmKXQkRERNqhkgiKqVrFkcB3gQnAccAi4BpgorvnKwn9HjgZOBg4FqgA3gXuBX7p7k/n6b25U1UFc+cWuxQiIiLSDpVEUARw9/nEsnvZXJt1TaK73wrc2si5m4Gbs31WSaquVtOziIiI5EUpjHqWttD0OCIiIpInCorlLhnM4k3ORy4iIiLSYgqK5a66GjZtgg0bmr9WREREpAUUFMtdVVXs1fwsIiIiOaagWO6qq2OvAS0iIiKSYwqK5U41iiIiIpInCorlLgmKqlEUERGRHFNQLHdJ07NqFEVERCTHFBTLnWoURUREJE9yGhTNrJeZVebymdIMDWYRERGRPGlxUDSzcWb2f2bWK+1YfzObDCwBlpnZVbkspDRBg1lEREQkT1pTo/gF4BR3fz/t2E+Bw4E3gaXAJWZ2eg7KJ81RjaKIiIjkSWuC4v7AP5NfzKwrcBrwqLsPBYYC84ELclJCaVqXLtChg2oURUREJOdaExT7AwvTfh8FdAFuBXD3VcBDRGCUfDOrX+9ZREREJIdaExTXA13Tfj8ccOCptGMrgd5tKJe0RHW1gqKIiIjkXGuC4n+Bo9J+PxWY5e5vpx3bmRjYIoVQVaWmZxEREcm51gTF3wH7mdnzZvY0sB9wZ8Y1BwEz21o4yZJqFEVERCQPOrXinl8Do4GPAQb8BfhxctLMDgH2Au7KRQElC6pRFBERkTxocVB0943AJ8zsgvjVMxPKW8CBwJy2F0+yUlUFb7/d/HUiIiIiLdCaGkUA3H1lI8eXoP6JhaWmZxEREcmD1qzM0svM9jazzhnHzzGzB8zszlTzsxSKmp5FREQkD1pTo/hD4CxiPkUAzOwLwNVEn0WAk8xspLu/1vYiSrNUoygiIiJ50JpRzzXA4+6+Lu3Yl4C3gSOAZOm+y9tYNslWMuF2XV2xSyIiIiLtSGtqFHcCHk9+MbO9iXkTv+ru/0wd+ygRGqUQkvWe166N0CgiIiKSA62pUewK1Kb9XkOszPJY2rHZRKCUQkjCofopioiISA61Jii+DQxL+/0YYsm+l9OO9QLSm6Yln5KgqH6KIiIikkOtaXqeBPyvmX2eqFk8Afiju6d3kNsdmJ+D8kk2kqZnBUURERHJodbUKP4IWA38AriRCItXJCfNrD/wIeDZHJRPsqGmZxEREcmD1qzM8l8z2wc4LXXoQXefl3bJYOBXbL3+s+SLahRFREQkD1q1Mou7vwP8spFz/wL+1ZZCSQupRlFERETyoNVL+AGYWQUxsKUnsAKYkVoLWgpJg1lEREQkD1rTRxEz625m1wPLgenAk8BLwHIzu97MeuauiNIsNT2LiIhIHrS4RtHMugPPAPsAq4CngUXAAOAA4DzgMDMb4+4rc1hWaYyankVERCQPWlOj+P+IkPhrYLC7j3X3j7v7WOoHsuyduk4KoaICOndWjaKIiIjkVGuC4inAc+5+kbsvTz/h7ivc/QvAFODUXBRQslRVpRpFERERyanWBMVBRJ/Epkwm1n/OmpkNNLNbzGyhma03szlmdrWZ9WpFGZNnHmFmm83Mzez7TVw3xsz+ambLzGytmf3bzC41s46tfXfBVVerRlFERERyqjVBcS3Qv5lr+qWuy4qZ7QZMA84BXgB+DrwFXAJMMbM+LS2kmVUDv2uuHGZ2IvAUcATwZ6LpfLtUGe5u6XuLpqpKQVFERERyqjVB8V/AR81sj4ZOpkLf6bRsLsXriPB5sbuf5O5fc/ejiLA2FPhBK8r5C6AHsZJMg1IDc24CNgNj3f3T7v5lYlDOFOA0MzujFe8uPDU9i4iISI61Jij+BKgC/mVm3zOzo8xsLzM70swmEgGxCvhpNg8zs12B8cAcojYv3XeANcDZZlaZbQFTtYTnABcDC5u49DSi9vNud5+aHHT3WuCbqV8/l+17i0pNzyIiIpJjLQ6K7v44cCHQBfg68CjwKvAY8C2gEvi8uz+W5SOPSu0fcfe6jHetIqbi6QaMzuZhqbWmbwLud/fbs3z33xs49xTRbD3GzDpn8+6iUo2iiIiI5Fhrl/C7wcz+BpwNHEg08a4gJt2+3d3ntuBxQ1P7Nxo5P4uocdwTeDyL591IBOAL2vJud99kZv8lpgLaFZiRxfOKRzWKIiIikmOtXsLP3efRSN9BM+sCbJflhNs9UvsVjZxPjje72ouZnQucCHzM3d/N97vN7DxignEGDRqUxevySDWKIiIikmOtWsIvC78GluXoWZbae5MXmQ0Brgb+4O73FuLd7n6ju49095H9+vXL0StbSaOeRUREJMfyFRShPmQ1J6m169HI+e4Z1zXmFmAd0X8yW7l6d/FVV8P69bBxY7FLIiIiIu1EPoNitmam9ns2cj6ZhqexPoyJg4gpdhanJth2M3Pgt6nz30gduz+bd5tZJ2AXYBMxp2NpS9Z7Vq2iiIiI5Eir+yjm0KTUfryZdUgf+ZyaNLuGqCl8rpnn3EaMjs60BzGZ9nRiUu+X0s49AZwJTADuyrjviNTznnL39dl9lCKqro796tXQq9WL2YiIiIh8oOhB0d1nm9kjxMjmi4Br005PJKbbucHd1yQHzWxY6t7X055zcUPPN7NPEaHvYXf/Zsbp+4AfA2eY2bXJXIqpwTjJkn+/bv2nK6CkRlEDWkRERCRHih4UUy4EngWuMbNxxFQ0o4AjiSbnb2Rcn0xVk20/yAa5+0oz+ywRGJ80s7uJQTgnEFPn3Afc05Z3FEx6jaKIiIhIDpRCH0XcfTYwEriVCIhfBHYDrgEOdfeleXz3/cCHiAm2TwW+AGwELgfOcPcmR1uXDPVRFBERkRzLqkbRzDbnuyDuPp9Ydi+ba7OuSXT3W4kA2tQ1zwDHZfvMkqSmZxEREcmxbJueW9PEWx41ce2Fmp5FREQkx7IKiu5eEk3U0gTVKIqIiEiOKQC2F6pRFBERkRxTUGwvuqWmkFRQFBERkRxRUGwvOnSAyko1PYuIiEjOKCi2J9XVqlEUERGRnFFQbE+qq2H58mKXQkRERNoJBcUy8dhj8MlPQl1dExcNHw7PPgtlMke4iIiIlDYFxTIxfz78/vcwc2YTF40fDwsWwIwZTVwkIiIikh0FxTJRUxP7Z55p4qJjjon9P/6R9/KIiIhI+6egWCb22AP69m0mKA4eDEOHwiOPFKxcIiIi0n4pKJYJMxgzppmgCFGrOHky1NYWpFwiIiLSfikolpGaGpg1CxYvbuKi8eNh3Tp4+umClUtERETaJwXFMpL0U3z22SYuGjsWtttO/RRFRESkzRQUy8iIEZEBm2x+rqyEww5TP0URERFpMwXFMtKlS4TFrPopvvIKLFxYkHKJiIhI+6SgWGZqamDqVFi/vomLxo+PvWoVRUREpA0UFMtMTQ1s2ADTpjVx0fDhsP326qcoIiIibaKgWGbGjIl9k83PHTpEreKjjzaz5p+IiIhI4xQUy0z//rD77ln2U1y6FF58sSDlEhERkfZHQbEM1dTEFDnuTVz04Q/HXs3PIiIi0koKimWopiYm3X7zzSYu6t8fDjxQQVFERERaTUGxDGXVTxGi+XnKFFi5Mu9lEhERkfZHQbEM7bUX9OyZZVDctAkmTSpIuURERKR9UVAsQx06RK1ik0v5QVxUWanmZxEREWkVBcUyVVMDr70Gy5Y1cdF228GRRyooioiISKsoKJappJ/ilCnNXHjMMfDWWzB7dt7LJCIiIu2LgmKZOuQQ6NQpy36KoFpFERERaTEFxTLVrVvMftNsP8Xdd4chQxQURUREpMUUFMtYTQ288AJs3NjERWZRq/jEE7FItIiIiEiWFBTL2JgxsG4dvPRSMxcecwysXg3PPVeQcomIiEj7oKBYxmpqYt9sP8WjjoKOHdX8LCIiIi2ioFjGdtwxuh8220+xRw8YPVpBUURERFpEQbHM1dREjaJ7Mxcecwy8+GIsEi0iIiKSBQXFMjdmDCxaBHPmNHPhMcdEmnzssUIUS0RERNqBkgmKZjbQzG4xs4Vmtt7M5pjZ1WbWqwXP+LKZ/TV172ozW2lmr5jZVWY2sJF7vImt5Ed/ZN1PccQI6N1bzc8iIiKStU7FLgCAme0GPAv0Bx4AXgcOAS4BJphZjbsvzeJR5wOrgcnAu0AFcCBwGfBpMxvr7g2NEZ4L3NrA8QUt/CgFt+++0L179FM866wmLuzYEY4+Gh55JGoWzQpWRhERESlPJREUgeuIkHixu1+bHDSzq4iQ9wPggiyes6+712YeNLPPAjemnnNcA/fNcfcrWlHuouvYMcapNFujCNH8fO+9se5fsgagiIiISCOK3vRsZrsC44E5wK8yTn8HWAOcbWaVzT2roZCYcm9qv0cri1nSxoyBV16BFSuaufDkk2Oo9LnnxryKIiIiIk0oelAEjkrtH3H3uvQT7r4KeAboBoxuwzuOT+3/3cj5nmZ2rpl93cwuMrO2vKvgamqiNbnZ+bR79YI77oA33oAvfKEgZRMREZHyVQpBcWhq/0Yj52el9ntm+0Az+4yZXWFmPzWzfwC/I/ohfq2RW/YHbiaapn8JTDGz6Wa2X7bvLKZRo6BDhyzmUwQYOxa+9S249Va4/fY8l0xERETKWSkExR6pfWMNp8nxni145meIZusvEs3a04C8MlI+AAAgAElEQVSj3X1WA9deBdQA/YBq4GDgPiI8PmFmOzX2EjM7z8ymmtnUxUWcn7C6GvbfP8t+ihBB8fDD4YILonZRREREpAGlEBSbkwzPbW5K6Q+4+2h3N6AvERQBppnZhAau/aK7P+vuS9x9tbtPdfePAn9M3f+lJt5zo7uPdPeR/fr1y/oD5cOYMdH0vGlTFhd36gR33gmdO8MZZ8D69Xkvn4iIiJSfUgiKSY1hj0bOd8+4LmvuvtTdHyXC4jrgNjPrmuXt16f2R7T0vcVQUwNr1sC/G+uFmWngwGh+fukl+MpX8lk0ERERKVOlEBRnpvaN9UFMRiq3uo3U3ZcDU4jm5X2yvC1pS252tHUpSCbezqqfYuL44+HSS+Gaa+DBB/NSLhERESlfpRAUJ6X2481si/KYWTXRf3Ad0NZVUpK+htk0zkL9KOu32vjeghg0KLbf/x42bGjBjVdeCQcdBOecA/Pn5618IiIiUn6KHhTdfTbwCDAEuCjj9ESiRu82d1+THDSzYWY2LP1CMxucmpNxK2Z2PjFIZT7wStrxgxqan9HMhhMjoAHKZmjwT34CL7wAn/98TJeTlc6d4e67I11+4hNZdnIUERGRbUGprMxyIbGE3zVmNg6YAYwCjiSanL+Rcf2M1D59HboDgT+Z2bOpe94F+hA1g/sRS/ud7e6b0+65GDjFzJ4gQuR6YBgwAegI3ATclaPPmHennw4vvww//GGMgr4oM3Y3Zo894IYb4MwzYeJE+N738lpOERERKQ/mWVc95ZeZ7Qx8lwhpfYBFwP3ARHdflnGtA6RGNifHBhFrQx8ODAZ6A7VE0/GjwC/cfX7Gc04CPgkMJ5YQ7AIsBaYCN7l71h33Ro4c6VOnTm3BJ86Pujo46ST461/hscdi2sSsnXtuDHB57DE46qhmLxcREZHyZGbT3H1ks9eVSlAsd6USFAFWroRDD4V334V//Qt22SXLG9esgZEjYenSmJRxj3a54qGIiMg2L9ugWPQ+ipJ73bvDAw/A5s1w4oktWNa5shLuvz+qJY85Bt55J6/lFBERkdKmoNhO7b473HMP/Oc/8L//G9kvK0OHRrv1u+/ChAmwosXTV4qIiEg7oaDYjo0fDz/9KfzpTy0cn3LIIfDHP0bKPOkkqK3NWxlFRESkdCkotnOXXho1ildcAX/+cwtunDABfvtbePJJOPvsaMcWERGRbYqCYjtnBtdfD6NGRd575ZXm7/nAWWfBz34G990HF1/cgskZRUREpD1QUNwGdOkSzc/du8fglrffbsHNl18OX/4yXHcdfP/7eSujiIiIlB4FxW3EjjvGgObFi+Hgg2MFl6xdeSV88pPw7W/DTTflrYwiIiJSWhQUtyGHHALPPhs1jEccAbdnuzhhhw7wm9/AscfCBRdE4hQREZF2T0FxG7PfflGbeOih0Wfxq1/NcpxKRQX84Q8xIfcZZ8Djj+e9rCIiIlJcCorboL594ZFH4HOfg//7v+i3uHJlFjdWVsLDD8ckjccdF1PoiIiISLuloLiNqqiI8SnXXQf/+AeMHg1vvpnFjX37wlNPwYgRcPrpcOONeS+riIiIFIeC4jbuc5+DRx+F996LPoxZtSj37g2PPRZzLZ5/PvzgB5o6R0REpB1SUBTGjo1+izvtFEs8X3VVFkv+desWg1rOOgu++c2Y2TvrdQJFRESkHCgoCgC77hojoo8/Hr74xQiPs2Y1c1NFBfzud3DZZXDNNREaN2woRHFFRESkABQU5QPV1TEx929/Gyu4DB8eC7M0OSq6Q4e46Mor4a674IQTYM2agpVZRERE8kdBUbZgBp/6FPznPzB+PHzpSzBmTPze5E1f/WrMtfjoozBuHCxdWqgii4iISJ4oKEqDkpVc7roL3noLDjooxqxs3NjETZ/+dFRJTp8Ohx8Oc+cWrLwiIiKSewqK0iizmFv7tdfg5JNjzMohh8BLLzVx04knxiSNixbFnDvTphWsvCIiIpJbCorSrH794O67o7LwnXdirehPfxr+/e9GbjjiiBgZ07lz/PzwwwUtr4iIiOSGgqJk7eSTo3bxggsiOO6/f4yO/vOfGxjwstde8NxzsT/hBPj1r4tRZBEREWkDBUVpkV694Je/hPnzY/m///4XTjklVvX72c9g+fK0i3fYASZPjuX+LrwQvvIVzbUoIiJSRhQUpVV694Yvfxlmz44lnwcNihHSO+0UmfDVV1OLtVRWRpXjhRfCT34CH/841NYWu/giIiKSBXMtvZYTI0eO9KlTpxa7GEX10ksx7/add8a82/37w2GHpbYa54BJP6fia1+MA/ffD336FLvIIiIi2yQzm+buI5u9TkExNxQU6733XuTAZ56Bf/4zpteBWPVv1C7vctjrN3PYDm8y9M7v0PvAwVRVxQhrERERKQwFxQJTUGzcwoX1ofGZZ+Cll5y6uvpk2KlT9H3s3bt+37t3VDjusQfsvXeMidl+ewVKERGRXFBQLDAFxeytWgXP37+Ied+8kWXzVvF+zfEs2/dwlr3fgfffh2XLYlu8GFavrr+vZ88IjHvtVR8e99sPBg5UgBQREWkJBcUCU1BshXXr4AtfgJtvhg9/ODo39u37wWn3qI2cMaN+e+212L/3Xv1jdtgBRo2KycBHjYKRI6FHjyJ8HhERkTKhoFhgCoptcPPNcNFFMfrlD3+ItNeMZcsiME6fDs8/H9sbb8Q5Mxg2LILjIYfEaoL77AMdNMZfREQEUFAsOAXFNnrxRTjtNFiwAK6+Gj73uRa3J7//PvzrXxEaX3gh9osXx7m+fWNy8LFj4cgjo9lazdUiIrKtUlAsMAXFHHj/fTj77Fjy78wz4YYbYh7GVnKPCcEnT4Ynn4RJk2KicIjKyyQ0jhgRoXHTpvpt8+Ytf99lF9h3X9VKiohI+6CgWGAKijlSVwc/+hF861vRfvzxj8NBB0Wa22GHNj06CY5JaJw0Cd5+O/v7+/aFo46Kbdw42G237Gsl16yB9etjVLdqMkVEpNgUFAtMQTHHHn0ULrssRq8kf0Z33DECYxIcR4yAAQNanbzcY47HV1+Fjh1jmp5kn2wdO0Yt4n/+A48/HtvChXH/oEH1oXH06Fi+cO5cmDcv9uk/L10a93TuHB9jwIDYJ1vye8+eUF0NVVX1+8xazM2bo2Z09uwo/+zZ9dt//xvPmTAhtsMPhy5dWvm/gYiItFsKigWmoJgnK1fGiJUXX4Rp02J7/fX68LjffnDttfChDxWkOO4xaObxx+GJJ6JWctmyra+rrITBg2MbNCj2nTvDokWxLVwY26JF8RGb0q1bhMbq6nj/vHmwcWP9+YqKaBrfddfYz54dze3r10PXrtG8ngTH3XdXjaaIiCgoFpyCYgGtXg0vvxwjV37xC5gzJ/o2/uQnMSt3AdXVRVGmTYum6SQctqSJefXq+gC5YkXMM7lqVRxPfk5+r6uLMLjbbhEMd9st5pHs2HHLZ65ZE2Hx73+PbdasOL7rrnD00TGR+U47xb0DB0YtZOfOjX/GpUvrw+3ChfDOO/H5OneOGsv0LTnWteuWNaPV1XE8m++lri6C7qZNEbrVN1REJLcUFAtMQbFI1q6NPo3/93+RTL7//RgxnZmctnGzZ8M//hGhcfLkhmsx+/WL0LjTTlFLmdR8Llq0ZQ1mW3TsGKExCY4QgbC2NvbJtmnTlvdVV8fcmOlb9+6xr6qK8m63Xf2W/ntFRTx/9eoI0OlbcsysPnjvvnt9EFezvYi0V2UXFM1sIPBdYALQB1gE3A9MdPf3s3zGl4Ejgb2BvkAdMBd4FLjK3Rc0ct/ewBXAWKB76p67gSvdfV0271ZQLLI33oi5GB97LPowXndd0/MxLlsGzz0HU6ZEavrylyMlbSNWroyZiBYsiAE96T/Pnx9BLb0PZWZ/yh12iHCVBLza2q1/Xrs2glhSM5q5X7WqvlYy2ZIayWTr1CmuW7kyalsb2tasiSC7YUN2n71Ll6ilTN82b44wnR6gzSI0J+GxsjLekb4l792wIWpB+/ffss9p+j4Jxtlw37rPa7JftQp23rm+S0OyTwJ+U+rq4n+XtWu3HN2fPso/+Tn5T0NSA5y+N4ta3sGDNbm9SLkqq6BoZrsBzwL9gQeA14FDiNA3E6hx96VZPOdNYDXwMvAuUAEcCHwIWAmMdfeXMu4ZBTyRuvY+YD5wFDASeAYY5+7rm3u3gmIJcI8Juy+7LKrBPvvZqG3s2TNm554yBZ59Nvavvx73dOwYW0UFTJwIF1/c/H9tpSS5R8jJDHEbN0boTEJhY5XN7tHEPns2vPnm1vv167estczcIFYMWrgwFh3KVFUVXRIqKhrfOnaMZv158yIQpuvcOUJhdXWE+vTViSCC2447Rnjr1Km+1nTt2vqfGypXW+20U0xon2x77x1bZoDcvDnC75Il8T0nG0Q/3K5dt9537Rrf7apVce/y5fGPg8yf16f+hk5CbObPENckIXnt2vgu0n83i38rDhoUQTwJ48m+T5/c9O/dsGHLz5D5mVatis+frHvfq9eWW/fuxeuKsWlT/Z/xhQvj58rKKGey9elT3DJK9sotKP4DGA9c7O7Xph2/CrgMuMHdL8jiOV3cvbaB458FbgT+5u7HpR3vCLwC7AWc6O4Ppo53AO4FTgX+n7tf2dy7FRRLyKpVcMUV0X8xGQGyYkWc69MHDj0UxoyJ/cEHx992F18MDz0Ug2N+/WuoqSnqR5DylfxxS2+6T/7Dunx5BNfGtk2boptt5kCoQYOitjI9qKxbF7W/DdU6bt68da1psnXrFltFxZYj/dNH/Cej/ZP/PKTvk583bowQ/Z//xDZjRtQkJ5K+r++/H4Hw/ffr782VDh0ikHbpsmXZMn+G+n6zyedP/7lbt/juFyyI73T+/K1rqLt2jb9OklDfqVPDQX/Dhi27UWRumd0qWvOZk3/wdOiw5ZZ+LClvdXUEt8yfKyujhjmpRU629N9XrKj/s/v22/Duu3FPNmVMgm6fPtF/O9k39HOvXvHv+Wz7MGcjPdQuWhT/AKutrf//Wfr/55Kfk89dV7flln6soqL+HzHJn6n039M/Q+Y/XJL9xo1b/iMlc1u3Di69NGbTyKeyCYpmtiswG5gD7ObudWnnqokmaAP6u/uaVr6jB7AceNPd90g7fhTwOPCUu38o456kXHOBXbyZL0pBsQT9+9/wwx/G30BJOGxs2K87PPhgBMZ58+Ccc6LfY9ra0yLSuM2bY1xZEhz/858IFr171weC9NCQbGZb1vBl7jdsiHDTs2dsPXrU7xuaPioX6upiVackNM6bF/vVqxsOGMnvmzZFDWh694lkS45XVtZ/lvTPk2yVlfHZ338/tmXL6n9OtlWr4q+shoJM8vu6dfXdNpKuHsnPzfU5Thpaqqujxrixbij9+8d7li2LbenS+p+T35NtyZLYmqrV3m67Lb+P5Ofq6gjmmcE4fVu9essBd9mG2qRBKQn6mQE8/fck5K1bV7/lKkJl/sPl5z+HY4/NzbMbU05B8TPATcCN7n5+A+eT2saj3f3xVr7jLOD3wJ/c/dS0498HvgF83d1/1MB9M4E9gd3dfXZT71BQbCfWrIHvfQ9+9rP4r9OPfwznnqt2FBFpN9avj7/qkmCUbEkYy6e1a7cOj5lN8JndC1aubLyGL/m9srLh/tTJzzvsEGEsvQa4U6e21WC61wfH2tr6fXKuoRp5iO86vXa/S5fi/Ccm26DYqRCFacbQ1P6NRs7PIoLinkTtX7NS4XMgUAXsBxxN1Ax+rRXv3jO1NRkUpZ2orIQrr4zpdi68MPo53nxzhMcPfUj9F0Wk7CW1nMWQhKOddy7O+3PJrL6Pcnse1FUK1STJ17uikfPJ8Z4teOZngO8AXyRC5jSiRnJWLt9tZueZ2VQzm7p48eIWFE9K3j77xFp/t90WHbE+/OFoZznzTLjnnuZnyRYREWkHSiEoNiepGM66jdzdR7u7EVPkjE8dnmZmE3L5bne/0d1HuvvIfv36tfDRUvLMomZxzhy4/344+WR45BE444zoaHXMMTENz4IGZ13K3saNMcHh5ZfH80VEREpEKQTFpNausYrb7hnXZc3dl7r7o0RYXAfcZmZdC/FuaUe6dYMTT4Rbbomhc08/DZdcEgsrX3RRtKGMGBFzMT74YMNr+mXavDlqLC+4IDrSHHssXH11hM/TT297+BQREcmBUgiKM1P7PRs5n4xSbqwfYbPcfTkwBegH7FPId0s707EjHHZYLBc4cya89lrM1ditG1xzTQTKPn1g+PAIkffcE0PwIHoyT5kSIXPgwFiE+fe/h/Hjo8Zy+fLoC/mXv8CwYTGgJldLooiIiLRCKYx63g14k6anx+kA9Gvt9DipZ70AHAwc6O7TU8c0PY7kTm0tvPACPPVUbM8+G0MLIZb32LQpJrnr3BmOOy6asD/ykRhAk+6tt2Kanocfhn33jXkdDzus8J9HRETarWxHPRe9RjE17cwjwBDgoozTE4FK4Lb0kGhmw8xsWPqFZjY4Fe62YmbnEyFxPjHBdmIyMAM4wsxOSLu+A/Dj1K/XNxcSRYCY4+CII+Cb34y+hsuXR3D86U8j8A0fDr/7XUzw9ac/RRNzZkiEWGT4L3+JWsaVK+Hww2Nex8ylOERERPKs6DWK0OASfjOAUcQSfm8AY9KX8DMzB0gNWEmOnQT8KfWcN4gl/PoAo4kpclYD/+PukzPenbmE3zxgHFrCT0rBmjXw/e9HM3RlJXznO/DRj8YsuCIiIq1UNhNuJ8xsZ+C7wAQi4C0C7gcmuvuyjGsbCoqDgEuAw4HBQG+gFngLeBT4hbvPb+TdexO1l0cC1URz813Ale6e1eqoCoqSVzNmRJ/HSZPi9333jYEvxxwTNY5duhS3fCIiUlbKLiiWOwVFyTt3eOUV+Mc/Ynv66VjfrGvXmAw8CY5Dh2olGRERaZKCYoEpKErBrVkDkyfXB8eZM+vPbbdd/Qr1mVvXrrFIbuZWXV3/8y67xNrYnUph8SYREcm1clrCT0Rao7IyRk8fd1z8PmcOPPpoTMdTW7vllixCmvy8aBGsXr3lVle35fN79oz5HY8/HiZMgF69Cv4RRUSkuBQURdqLIUNiberWcI8AuXo1rFoF06fHyOuHH4a77or5Iw8/PELj8cfDHns0/8z0Z69eHeF00aKYtHzRohj9feSRMY+kiIiUJDU954ianqVd2rwZnn8+QuNDD8Grr8bxPfeEHXaIn83q9+k/r19fHwrXrt362WYRIs87L0Z1V1Xl//OIiAigPooFp6Ao24T//jcC4yOPRC1h8veH+5Y/A1RURJjcYYdYpnDAgPqfd9ghms6//e2YZ3LXXWOVmkMPLc7nEhHZxigoFpiCokgrPfUUfPKTMH8+fP3rER4rKopdKhGRdq1sVmYRkW3cEUfAv/8dYfH734fRo2PeyEJZuBD++leYN6++NlRERAAFRREpBd27w29/G0sbzpsHBx0E11yz9UjsXHrxRTj77BgE9JGPwODB0L9/zEX5//4f/OEPMHu2wqOIbNPU9JwjanoWyZF33onR2w89BDU1sNtuMTBm/fqY3ifz5379YNw4OPpoGDmy6bkf6+riuVddFXNQVlXBpz8NJ5wAr78e4fHFF2PQzsaNcU+PHnDggTBiRGwjR0aZNKm5iJQx9VEsMAVFkRxyh9/8Bn70owh3nTvHZOGdO2/985w58NJLcU+PHnDUUfDhD8e2224xunr1arj1VvjFL+DNN2HQILj4YvjMZ+KeTOvXw3/+A9OmRXCcNi2ax9enln3v3n3L4DhiRP27RETKgIJigSkoihTRkiXw+OMx4fijj0bzNURz8qhRMUp7+fL4+fLL4ZRTWr7qzMaN9eFx6tTYv/xyLKMIER4HD4add44guvPOW24DB0awFREpAQqKBaagKFIi3GHWLHjssQiNU6bEZOGXX5776Xc2bIjwOHVqhMZ582L09vz5sHTp1tcfcEDUZH7iEy0PjS+/HDWi06fH5xg7Ntb47t8/Jx9FRLYtCooFpqAoIltYu7Y+NM6fD3PnxmCdV16B7beHCy+Ez30u+lg2ZvNmePDBCIiTJ0O3btHUPW1arPUNsNdeERqT4Lj99oX4dCJS5hQUC0xBUUSa5R5N5D//eUzJ07lzjLy+9FLYZ5/665Yvh5tvhl/+MvpgDhoEn/989Kns1SuawV98EZ58MrZ//jP6YQIMGxY1l4MHx4juZBs0KIJmU2pr490rVkDv3k2HWBEpawqKBaagKCItMmNG1BT+7ncR0I45JoLgk0/GwJs1a6LJ/JJL4MQTm+5TmQTHyZNjmzkzmsGTkduJ/v0jNA4YEM9fsaI+GC5fXt/fEmJgziGHxNRB//M/ET7bOlinri6a5NPX++7XL56dLAlZaEuWwAsvxPerdcdlG6KgWGAKiiLSKkuWwA03RO3hO+/AdtvBGWdEQDzooNY/d/PmCGNz50atZLKfMyfeU1UVI7579ICePbf++a23Yiqhf/0rakJ32ilC40c+EtMRVVbWv8s9wuaCBVtub79dHwrfeSeC4aZNDZd3hx1iGqJkO+CAWNoxmYaori6+q7ffrt+Sd5hFAN5ll9iSMJw5hdGGDdHH8/nnY3vuuZgrM3H55fCTn2jqI9kmKCgWmIKiiLTJ+vVRG7j//qXVz/Ddd+Fvf6tf43vVqmgyP/zwCG9JKFy7dsv7zKIGM32d7/S1vgcMiPOLFsX0RtOnx/611+rDZHU17LFH1EIuXLh1DWmHDvGsuroIouk6d65vft9xx5gn86WX6qc4GjAgVgEaNSr2990XYf3jH48a3e22y8e3KVIyFBQLTEFRRNq9DRvg6acjNE6aFH0eBw5seBswoHVrdtfWxkjyJDi++WY0T++005bbwIERqJMm+XXr6mtN//vf2JKfFyyA3XevD4WjRsX96U3p7vDjH8eqPOPGxcCj7t1z8a1lZ+PGCKg33BDB9tBDYzv44C1rb0VyREGxwBQURUTagdtui9V69t03BhwNGJDf923eDHfcARMnRnP//vtHWJ45M8537Aj77VcfHA89NCZ3r62FlSujyX/lyi23Vaui+f7QQzUJvDQq26DYwhlnRURE2rFPfjKaxE87DcaMgb//HYYOzf176upiPfErrohm8QMOgL/8JfqAmkVz+/PPxzygU6bA738Pv/513NuhQ3broO+3H5x/Ppx1VsMrEDVk6VK4//4o25Il9aPmk76fyb65EfTSbqhGMUdUoygi0o5MnQrHHRc1fg891PBk7Zs2xWjzSZPgiSeiyXzXXaM2Mn3r3bv+HveYG/Pb345lIffZJ2oTTz656UE0mzfH8597LprUu3eP8Ne9+9Y/d+kCDz8M118f5evWLSZ5P//8mIcz0/vvRzi8996YqH7Tpqi13H33+gFQSd/ORP/+8VmPOCJGxR96aMtXO5KiUtNzgSkoioi0M2++CRMmxECae+6J2r6XX64Phk8/HU29AHvvHc29c+bAq69Gk3BiwIAIjPvsE3NeTp0ag3SuuAI+9rFoXs6XqVMjMN51Vww4GjEiAuNxx0UovPfeWMFo48aoLTz99NgOPLC+2bquLgY1JX0+k/3MmVHbuWlTzO95zDHxHU2YAH375u8ztca770aQ7dkzv993GVFQLDAFRRGRdui99yL8vPhi1Nq9/34c32MPOPLI2MaO3XIeSPeYtufVV7fcXnstBuB8+9sx0Xoha+BWrIDbb4/Q+Oqr9ccHD64PhyNGtLxP44oVETQffjj6dL73XtSMjh4d39uhh9Y3lSfb5s1b/t6xYwx8amyrro4BPi0t25IlEfBvuy3myoR4Rq9eUcvbp0/91rt3/O9bURH/uzS2HzQomvTTa4nLlIJigSkoioi0U6tXw1e+EgNIknA4cGDLn1NXF0GlmANM3KMWcPJkOOqomFQ9V+Wpq4vlJR96KILjtGm5eS7AzjvXL1U5dmzUfjZU7vXr49233RbBdePGGCB0xhnQtWv0wVy2LPbp27JlMQgoWzvuGIEx2fbdN2qVu3TJ0QfOPwXFAlNQFBERSbNoUdSiduiw5daxY/3PZlHDuHHj1tuGDbFfsiSa7J98EhYvjmfvvHOsbZ4Ex/feiwE/d98dtb477ABnnhk1t/vvn1153evLsmlTbOk/r18fI9NfeaV+e+21+v6bHTpE385kyczMbeDAkgqSCooFpqAoIiKSR+6x9GWyxnl6cISoMTz55Bi5Pm5cYZr2N22CWbPqg+OMGbF85rx50S8y0/bbR9N3ZnjO3L73PTj66LwWXdPjiIiISPthFs27e+8NF14YwfH11yMwdusWIbGQk6RDhNG99ort9NO3PFdbG5O9J8Ex2VaujLKn99PM3FozWX2eKCiKiIhI+TGrD2mlqEuXmGJo992LXZI20crnIiIiItIgBUURERERaZCCooiIiIg0SEFRRERERBqkoCgiIiIiDVJQFBEREZEGKSiKiIiISINKJiia2UAzu8XMFprZejObY2ZXm1mvLO+vNLMzzexOM3vdzNaY2Sozm2pmXzSz7Rq5z5vYnsvtpxQREREpHyUx4baZ7QY8C/QHHgBeBw4BLgEmmFmNuy9t5jGHA7cDy4BJwP1Ab+B44KfAKWY2zt1rG7h3LnBrA8cXtPzTiIiIiLQPJREUgeuIkHixu1+bHDSzq4DLgB8AFzTzjHeAs4A/uPuGtGdUA08CY4CLgJ81cO8cd7+iDeUXERERaXeK3vRsZrsC44E5wK8yTn8HWAOcbWaVTT3H3ae7+x3pITF1fBX14XBsLsosIiIisi0ohRrFo1L7R9y9Lv2Eu68ys2eIIDkaeLyV79iY2m9q5HxPMzsX2AFYAUxzd/VPFBERkW1aKQTFoan9G42cn0UExT1pfVA8N7X/eyPn9wduTj9gZi8DZ7v7K618p4iIiEhZK4Wg2CO1X9HI+eR4z9Y83Mw+DzAncbwAAA4USURBVEwApgO3NHDJVcAfiaBaCwwDvgqcBjxhZge4+9uNPPs84LzUr6vNbGZrytgCfYEleX7Htkjfa37oe80Pfa/5oe81P/S95kcuvtfB2VxUCkGxOZbae4tvNDsFuJoY6HKqu2/MvMbdv5hxaCrwUTO7DzgV+BIxoGYr7n4jcGNLy9VaZjbV3UcW6n3bCn2v+aHvNT/0veaHvtf80PeaH4X8Xos+mIX6GsMejZzvnnFdVszsJOBu4D1grLu/1cJyXZ/aH9HC+0RERETahVIIiklz7Z6NnN8jtW+sD+NWzOyjwB+Ad4EPuXtrmoQXp/ZNjrYWERERaa9KIShOSu3Hm9kW5UnNgVgDrAOyGoVsZp8A7gIWEiFxVivLNTq1b2lNZD4VrJl7G6PvNT/0veaHvtf80PeaH/pe86Nw3d7cW9z1L/eFMPsHMbK5sQm3b3D3C9KODwNw99cznvO/xICVucCR7j63mfceBMx09zUZx4cDTwB9gDPd/c42fDwRERGRslQqQTFzCb8ZwCjgSKLJeUz6En5m5gDubmnHjgQeI2pJbwHmN/Cq5e5+ddo9twKnEKFwPrCeGPU8AegI3ASc76XwJYmIiIgUWEkERQAz2xn4LhHS+gCLiPWaJ7r7soxrGwqKnwJ+28xr5rr7kLR7TgI+CQwnQmoXYCkx8vkmd3+wTR9KREREpIyVQh9FANx9vruf4+4D3H07dx/s7pdkhsTUtZYeElPHbk2ON7ENybjnfnc/xd13d/fuqfcOcPfjSyUkmtlAM7vFzBaa2Xozm2NmV5tZr2KXrdSZ2Wlmdq2ZPW1mK83Mzez2Zu4ZY2Z/NbNlZrbWzP5tZpeaWcdClbuUmVkfM/uMmf3ZzN40s3VmtsLM/mlmn87sZ5x2n77XZpjZj83scTObn/pel5nZS2b2HTPr08g9+l5byMzOTv1d4Gb2mUau+R8zezL1Z3u1mT2f6tokKan/Fnkj2zuN3KM/r1kys8PN7I9mtij13/5FZvaImR3XwLV5/V5LpkZRttZAk/zrwCFEk/xMoCa9SV62ZGbTiVV3VgMLiG4Fd7j7WY1cfyIx+XotcA+wDDieWD3oPnf/aCHKXcrM7ALg10SN/yRgHrA90YWjB/H9fTS9u4a+1+yY2QbgReA1YlqvSmJQ3UhicN5od5+fdr2+1xZKtVy9QnQtqgI+6+6/ybjm88C1ROvSPcAGYgGGgcDP3P1LBS10iTKzOf+/vbuPkasq4zj+/VFsAS0tyMuCooCUF0EBIxRK0y4vIoQUaFJEEagIf1QNBCIqIUWWIKYGEVEkGBUJIhQkQQV5ESm0UFFsbAUEKS9dgrwVsJRCaUnh8Y9zRm7He3dm2t2Z3e7vk9yczLnn3p55enfmmftyDmkijB+WrH4jIr5f197Ha5MkzQQuJA2ofSvp83YrYF/gnoj4ZqHtwMc1IrwM0gW4kzTQ+Ol19T/I9Vd2uo+DeSEl1ONIg7Z355hdW9F2c9KX82rg04X6TUjJegCf7/R76vRCmpt9CrBRXX0XKWkM0uD2jmvrsd2kov6iHKcrHNf1iq9I97E/BVycY3RaXZsdSV+4rwI7Fuq3AJ7M2xzY6fcyGBagF+htsq2P1+bjelyOx13A6JL172t3XAfNpWdbm6SdSU+C9wI/qVt9PvAmcJIkj/NYISLuiYgnIv/lNDAN2BqYHRELCvtYBczML78yAN0cUiJiTkTcEhHv1tW/yHuD1HcXVjmuTcoxKXNjLscV6hzX1p1B+qFzCunzs8yXgVHA5RHRW6uMiGXAd/PLGSXbWd98vDYh37rzPWAlcEJErKhvE2vPMNeWuA6FKfyGq0Ny+ceSL+UVkuaTEskDgLvb3bkNUC3ed5Ssm0f6w50gaVRErG5ft4aU2gfYmkKd47r+puTyoUKd49oCSXsAs4DLImKepEMqmvYV19vr2hiMknQi8BFS8v0QMC8i3qlr5+O1OROAnYCbgGWSjgL2Ip3lfjAiHqhr35a4OlEcvHbLZdWMNE+QEsVdcaLYHyrjHRFrJC0B9gR2Jg3fZAWSNiaNIABrf2g5ri2SdDbp/rkxpPsTJ5K+gGcVmjmuTcrH5q9It0ac26B5X3F9QdKbwIclbRYRK/u3p0NSFym2RUsknRIRcwt1Pl6bs18uXyLdr/yJ4kpJ84BpEVGbOa4tcfWl58GrNvd11RzXtfqxbejLcOB4r59ZpF++t0XEnYV6x7V1Z5NuLzmTlCTeARxe+HIAx7UV3yY9BPCliHirQdtm4zqmYv1w8kvgUFKy+H5SUvNT0n2et0vau9DWx2tztsnlDGBT4DBgNOmz9U5gEml64pq2xNWJ4tBVGx7Ij623h+NdQdIZwNdJT+Wf1OrmuXRcs4joijT8VxfpafKdgYVKM0k1y3EFJO1POot4Scllu3XaZS6HdVwBIuKCfM/ySxGxMiIeiTSD2g9ISU5PC7tzXJPacDYinTm8OyLeiIh/AlNJo3dMlnRgk/vrl7g6URy8Gv1y3byuna0fx3sdSPoacBlpSJeD4//HPXVc11H+Ar6ZdIvJB4FrCqsd1wYKl5wXA+c1uVmzcX19Pbq2oas91DapUOfjtTnLcvl0RPyjuCKfDa9drdk/l22JqxPFwevxXO5asb72BGTVPYzWmsp45y+cnUgPaTzdzk4NZpLOBC4HHiEliWWD7Dqu6ynSnPWPAntK2ipXO66NfYAUnz2AVcUBoUmX9gF+lutqYwH2FdftSJdY/+37E/u0NJfFETl8vDanFqfXKtbXEslN69oPaFydKA5e9+Ty8PrZLiSNBg4C3gL+0u6ObaDm5PKIknWTgM2APw/zJ/L+R9K3gEuBRaQkcWlFU8e1f2yfy9rTpI5rY6uBX1QsC3Ob+/Pr2mXpvuJ6ZF0bK1e7LFpMTny8NmceKbEbJ2lkyfq9ctmby/bEtdODS3rpc+BND7jdf7HspvGA2y/jAWGbieV5OR4LgC0btHVcm4vp7kBXSf1GvDfg9nzHtd/i3UP5gNs74QG3m4nfnmV/+8BHSSNyBHBuod7Ha/OxvTbH4zt19Z8B3iWdbRzbzrh6Cr9BrGQKv8eA8aQZRxYDE8JT+FWSdCxwbH7ZBXyW9Cv3vlz3ShSm48rtbyJ9UcwmTYV0NHkqJOBzMcz/YPJ8t1eTzmz9mPJ7X3oj4urCNo5rA/ky/sWkMwpPkRKVbYHJpIdZXgQOjYhHC9s4rutIUg/p8nPZFH6nAz/CU/hVyvE7h3TlawmwAvgYcBQpSbkNmBoRbxe28fHaBEnbAPOBXUjfVQ+SEvCppMTvhIj4TaH9wMe109mzl4a/LnYgDUPwAukD6xnSwwN9nsnxstZZg6qlt2Sbg0gfcstIl/YfBs4CRnT6/QyGpYmYBnCv49pyXPcizcC0iDS/6xpSEv63HPPSv3fHdZ3jXTuOT6tYPwWYS0qA3sz/D9M73e/BspB+wFxPGungNdJg+y+Tpp07GdJJqJLtfLw2F98tSVcOl+Tv/VdJJ4sO6ERcfUbRzMzMzEr5YRYzMzMzK+VE0czMzMxKOVE0MzMzs1JOFM3MzMyslBNFMzMzMyvlRNHMzMzMSjlRNDMzM7NSThTNzIYJST2SQlJ3p/tiZkODE0UzsyblJKvR0t3pfpqZ9ZeNO90BM7Mh6II+1vW2qxNmZgPNiaKZWYsioqfTfTAzawdfejYzGyDFewIlTZe0UNJbkpZKukpSV8V24yRdI+k5SW9Lej6/HlfRfoSkGZLmS1qe/40nJf28j22mSXpQ0kpJ/5E0W9KH+vP9m9nQ5zOKZmYD7yzgcOAG4A5gInAK0C1pfES8XGsoaT/gT8Bo4PfAo8DuwBeBYyQdGhELCu1HAn8ADgOeBa4DXgd2BKYC9wNP1PXnq8DRef9zgfHA8cDekvaJiNX9+ebNbOhyomhm1iJJPRWrVkXErJL6I4HxEbGwsI9LgTOBWcCpuU7ANcDmwIkR8etC++OB2cC1kj4eEe/mVT2kJPEW4LhikidpVN5XvSOA/SLi4ULb64AvAMcAN1a+eTMbVhQRne6DmdmQIKnRB+byiBhbaN8DnA9cFRGn1u1rDPAMMAoYGxGrJR1EOgP4QERMKPn37yOdjZwcEfMkjQBeBUYCu0TE8w36X+vPRRExs27dwcAc4JKIOLvB+zSzYcL3KJqZtSgiVLGMrdhkbsk+lgOLgE2APXL1p3I5p2I/tfp9c7k7MAZ4qFGSWGdBSd2zudyihf2Y2QbOiaKZ2cB7qaL+xVyOqStfqGhfqx9bVz7XYn9eK6lbk8sRLe7LzDZgThTNzAbethX1taeel9eVpU9DA9vVtaslfH5a2cwGhBNFM7OBN7m+It+juA+wCngsV9cedumu2E+t/u+5/BcpWfykpO37o6NmZkVOFM3MBt5Jkvatq+shXWq+vvCk8nzgcWCipGnFxvn1JGAx6YEXIuId4ApgU+DK/JRzcZuRkrbu5/diZsOIh8cxM2tRH8PjAPw2IhbV1d0OzJd0I+k+w4l56QXOqTWKiJA0HbgLuEHS70hnDXcDjgVWACcXhsaBNJ3geGAKsFjSrbndDqSxG78BXL1Ob9TMhj0nimZmrTu/j3W9pKeZiy4FbiaNm3g88AYpeTs3IpYWG0bEX/Og2zNJ4yNOAV4BrgcujIjH69q/LekIYAZwMjAdEPB8/jfvb/3tmZklHkfRzGyAFMYtPDgi7u1sb8zMWud7FM3MzMyslBNFMzMzMyvlRNHMzMzMSvkeRTMzMzMr5TOKZmZmZlbKiaKZmZmZlXKiaGZmZmalnCiamZmZWSknimZmZmZWyomimZmZmZX6LwYYMDNGS9eZAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(1, 1, figsize=(10,6))\n",
    "ax.plot(np.sqrt(lr_model_history.history['loss']), 'r', label='train')\n",
    "ax.plot(np.sqrt(lr_model_history.history['val_loss']), 'b' ,label='val')\n",
    "ax.set_xlabel(r'Epoch', fontsize=20)\n",
    "ax.set_ylabel(r'Loss', fontsize=20)\n",
    "ax.legend()\n",
    "ax.tick_params(labelsize=20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAooAAAGJCAYAAAANCBK/AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3Xm8nPPd//HXJ4ucLCcRWUhFFglChCCWCmqpWGqppe6W0mrDbemt3LRV1NreP124lbtqKbVWi1ClVIoElSixiyBCSCKySeLknGxyPr8/PjPOZDJzZuZkZs7Myfv5eMzjOue6rrnmO1dOzrzPdzV3R0REREQkXbvWLoCIiIiIVCYFRRERERHJSEFRRERERDJSUBQRERGRjBQURURERCQjBUURERERyUhBUUREREQyUlAUERERkYwUFEVEREQkIwVFEREREcmoQ2sXoK3o3bu3Dxo0qLWLISIiIpLTSy+9tNDd++Q6T0GxSAYNGsSUKVNauxgiIiIiOZnZh/mcVzFNz2bW38xuNbOPzWylmc00s2vMrGeB1znKzJ4ysyVmtsLMppnZxWZWk+X8jmZ2rpm9amYNZlZnZpPM7NvFeWciIiIi1akiahTNbAgwCegLPAS8DewG/BA42MxGu/uiPK5zBXARsAwYBywC9gIuA8aY2YHuvjzl/I2Ax4D9gZnAbYlDhwJ3mtnO7v7fxXiPIiIiItWmIoIicD0REs9y9+uSO83sauAc4BfAac1dwMx2Ai4ElgC7uPv7if0GXAv8APgJcGnK084gQuJk4EB3r088pyvwFHCOmf3N3Seu/1sUERERqS7m7q1bALMtgRlEjd4Qd29MOVYLzAUM6JsMclmucznwM+A37v6jtGO1wFJgAfAld1+T2P8MsDdwmLv/Pe05hwEPAw+4+zG53seoUaM8vY/i6tWrmT17NitWrMj19KpWU1ND//796dixY2sXRURERPJgZi+5+6hc51VCjeL+ie341JAI4O51ZvYcMAbYA3iymetslti+n34gcZ2FRK3lCODVXM9J2XdAzneQxezZs6mtrWXQoEFExWbb4+4sWrSI2bNnM3jw4NYujoiIiBRRJQxm2SaxfTfL8emJ7dY5rrMwsV0nrSRqFHsnvh2Wz3OALRPbHma2WYbjOa1YsYJevXq12ZAIYGb06tWrzdeaioiIbIgqISj2SGyXZjme3L9xjus8ktiONbNBacd+TjRfA6SOok4+5wIz65zcaWZdiP6OZHhOQdpySEzaEN6jiIjIhqgSgmIuyRTSbGdKd58E3EiEutfN7I9mdpWZPU8MZJmaOHVNytN+C7wMjAammtn/mdnvEuduBXyc4TlNBTM71cymmNmUBQsWtOCtldaSJUu4/vrrC37eoYceypIlS0pQIhEREakmlRAUkzWGPbIc7552XlbufhrwfeAt4DhipPQq4CDgjcRp81POrwf2IWocVwGnAN8CphB9IpP3J2MKdPeb3H2Uu4/q0yfn5OZlly0orlmTMfd+4dFHH2XjjXNV4IqIiEhbVwmDWd5JbLP1Qdwqsc3Wh3Et7n4rcGv6fjP7Q+LLF9POrydGS/8s7fzBxGCX99x9cT6vXWnOP/98ZsyYwciRI+nYsSPdunWjX79+vPrqq7z11lt8/etfZ9asWaxYsYIf/vCHnHrqqUDTKjPLli3jkEMOYa+99mLSpElsvvnmPPTQQ3Tu3DnHK4uIiEhbUAlBcUJiO8bM2mWYHmc0sBx4vqUvYGZjgIHA0+4+J8+nnZLY3t3S113L2WfDq6/mPq8QI0fCNddkPXzllVfy5ptv8uqrrzJx4kS+9rWv8eabb34xOvnWW29lk002Yfny5ey6664cc8wx9OrVa61rTJ8+nXvuuYebb76Z4447jnHjxvHtb2vRGhERkQ1Bqzc9u/sMYDwwCDgz7fBlQFfgjtQ5FM1smJkNSzsXM+ueYd8Q4Cain+H5eT7nUOBcYA7Rj7FN2G233daawubaa69lxx13ZI899mDWrFlMnz59necMHjyYkSNHArDLLrswc+bMchVXRESkecuXw7//DW+8AZ9/3tqlKZ5WnuM6VSXUKEKskDIJuNbMDgCmAbsD+xFNzhemnT8tsU0fbnuLmQ0EXgIWA0OBw4GOwFh3z1Qr+baZvU4sG7gSGEXM7bgAOLxozc7N1PyVS9euXb/4euLEiTzxxBNMnjyZLl26sO+++2ac4qZTp05ffN2+fXuWL1++zjkiIm1eXR107gwdWvCx+ckn8OSTMHEiDBkCZ54JtbVFL2LVcYcPP4T27aFPH6ipaf78xkZ4+2144YUIhy+8AK+/3hQQa2pgxx1h551hl11iO3w4bLRR9muuWQNLlsDixRE627Vr/tGxY1wv+ejYEVo684c7zJoF06bF4+23m76eOBG2265l1y2yigiK7j7DzEYBlwMHE2stzyWW3rvM3T/N81KPAKcSA1lqiYEr44BfufvrWZ5zd+I19yQC5UfAVcAv3b3yhjIXoLa2lrq6uozHli5dSs+ePenSpQtvv/02zz/f4pZ9EZG2a8oUuPpquPfeCAYjRzaFkF12gW23jbCQqq4OnnkGnngiHm++Gfu7d4fPPoOrroLzz4czzojwWQ7uEUSeeioe06fDl74EW2yR+dGlS2nKUV8PEybAY4/Bo49CaitVbW0Exr59Y5v82h1efDEeyc+07t1h113hRz+K7fLl8NJL8PLLcPfd8Pvfx3kbbQQjRkToWrEiAuGnnzZtl+YcJ5tbhw5NwbFTJ+jaFbp1i0fy69TtJ5/Ev8Xbb8f9SOrZM36ejjhi3Z+pVlQRQRHA3WcBJ+d5bsb47u63A7cX+Lo/An6U88Qq1KtXL0aPHs32229P586d2XTTTb84dvDBB3PDDTewww47sM0227DHHnu0YklFpOosX16+kFNujY3w8MMREJ95JgLMmWdGzdHLL8Ntt8H//V+c26lTUy1Wr17w9NPw/PNRy1VTA3vvDSeeCF/9aoTMKVPgZz+D886LwHjhhTB2bFyn2D74oCkYPvVUBBSAAQMiPM2bF33n581b97mbbBLBJVlrllqLlnzU1EC/fk3hcsCA2Pbrt3bN63vvRSh89NGoKVu5MoLoV78a92GjjWDBApg/P7YLFsBHH0Xwm5+YqGTHHeM+7rZbPLbZJmr4Uh1/fGwbG+H995uC40svRTjt2jXe12abRXDs2bPpfW6ySfw8u8fzMz3WrIHVq+OxatXaj+S+FSsi/C1bFo+6Opg7N75O7u/dOwLh2LEwbFh8ve22EYwrcF7iVl/rua3ItNbztGnT2HbbbVupROW1Ib1XkQ3Wp59Gzdodd8DkyTB0aHzYf/WrsN9+8WFbzRoaIgRec03UuA0YAD/8YXygd0/pzt7YGMeTQST5+OwzGDWq6Z7suWf25tRnnoGLLoJnn43XufhiOOmkdWuSFi1qun7y9ebPj6DVuXNs07/u0CEC6QcfxDU23RT237/pMXjw2oFk5UqYMyfC2axZTY+6uuyBaNWquF9z5jTV8iW1a9dUW7lwYdwriHB3yCFw6KGwzz75hWP3CGgtafKXZuW71rOCYpEoKG4471Vkg7JyZdQE3Xkn/P3vERCGD4fDD4epU6OGqK4ugscuuzSFpNGjc/c5a4k1a6JmbM6cpsfKldHPrUOHzI/27Zvvd2YWwe33v48wvOuucO65cMwx+QeUxsaoTSqkydY9mqYvuij62w0dCuecE82iyWD44YdN5w8eHDWXW2wRNboNDU3b5GP58ijH9ts3BcNtty1tTdXSpWsHzOTjo4/ifhx8cATEIUNKVwYpmIJimSkobjjvVaTNc4/m0zvvhL/8JcLTpptG096JJ0YTajJ4rF4dfceS/fEmT25qdt1tt2jiHDEigsv220OPbGsrJKxaFTVh06fH44MPYPbsplD4yScRFovNDI48MgLi6NHlbQJ0h0ceiSbp116LfUOHrt0Xcqedqr/GViqKgmKZKShuOO9VpCTco69Yrik+amtzh61MFiyIEPfcc/DWW001TytWRI1c8usVK+JYfX00Z3796xEODzwwv9q1Zcuidu6f/4xasjfeWLtpcsCACIwjRkRN15Il0YctGQxnzozauaTu3aF/f9h886ZH+vddusR9W7MmtpkezfU9a2yMaw4cWPh9LabGxqilHTCgZf/GIgXINyiq0V9EpNxWr44Rj6++Cq+8EttXX40mx3xsumlTB/jUx5e+FDVhjY0xxcakSREMJ01q6ifWsWOcW1sbtX49esS2pib6jCW3O+4IRx+9dt+8fHTrFn3QDj00vnePJsg33ojRv2+8EY9//jPuA0RZttoqmnyPPz6+Tj569arIDv4l0a5dBGiRCqKgKCKVyb0pOKX2NevQoalfWWtqbIyQ9/e/R7PhK69ESOrRAzbeOPP2k08iEL7xRtTiQQSzESPgG9+ImrZcI4k//bRpvrU//Wnt6T1qa6PJ8oMPoqYOYoTlnnvGgIw994xmzHKOVjaLmrqBA+Gww5r2r14dI1M33jimQGntf08RyUhBUUQqi3tMTnzZZfCvf2U/Lxke+/Zdu1YtOd1Enz7FL9uyZVG2Rx6JgDh3bgSc3XeP0bErV0ZwW7o0gtqHH8Y2ua9nz+hr9oMfxHbkyBgJ2tIRne5Nc7IlJ+qdPj1G3u65Z/S1Gzq0MkNYx47x3kWkoikoyhe6devGsmXLWrsYsqFyh8cfh8svj750m28OV1wRTZ/p/c2SfdFWr4aPP46A9OyzMeozqVevpvA4YkQ0pe6wQ9Rg5VueTz6J/nyvvx5lmzAhBlt07w4HHQRf+1qM5uzbN/f1GhsjsBUztJnFnHX9+sX0NCIiRaagKCKtyz2mX7n88hj8MGBATFNy8smFTULc2Ni0HFZqDdsDD8DNNzedN2BAU2hMbrt3j0A4dWrTdurUtfsMbr11TLp82GGw117NLwuWSfrkwCIiVUBBsQ37yU9+wsCBAznjjDMAuPTSSzEznnnmGRYvXszq1av5+c9/zpFHHtnKJZUNknusfnH55TFf3KBBcNNN8J3vFB7CIIJYsi/cwQev/Tpz50at4GuvNT0efTTzNCs9e8Y8gccdF6s3DB8ej802a/FbFRGpVpoep0hyTY9z9tnRh72YRo6MBQSyeeWVVzj77LN5+umnAdhuu+34xz/+wcYbb0z37t1ZuHAhe+yxB9OnT8fM1qvpWdPjSFbLlsXgivTH1KkwY0ZMwnvhhfDtb5d3fdMVK6L28LXXoozbbtsUCCuxT5+ISBFpehxhp512Yv78+Xz88ccsWLCAnj170q9fP8455xyeeeYZ2rVrx5w5c5g3bx6bqbakbXOPUbl/+lOsRbtiRfNLc9XUxGCQPn2i/136dpNNYq69urpYtiy5Tf16/vwIhAsXrl2Wrl1jhYnttosJhk84oXWW56qpicmMd965/K8tIlIlFBTLpLmav1I69thjuf/++/nkk0/45je/yd13382CBQt46aWX6NixI4MGDWLFihWtUzgpvXfegXvuice770aN3d57Rz+9jTZa+9GxY9N2+fKYoHn+/Bgs8uqr8f2qVdlfq3PnmJ6le/d4bLJJzMM3ePDaj969VWMnIlIlFBTbuG9+85uccsopLFy4kKeffpp7772Xvn370rFjRyZMmMCHqeuISmVbsiSaa7t1iyCWDGXp/flmz4Y//znC4csvRyjbd1/40Y8iuLV0GTD3qC2cPz/m8uvSpakMtbWtUysoIiIlpd/sbdzw4cOpq6tj8803p1+/fpxwwgkcfvjhjBo1ipEjRzJs2LDWLqLk0tAA114Lv/xl0yTKqTp1agpsnTrFiF/3WOXi6qvhP/4jVuxYX2ZNtYUiIrJBUFDcALzxxhtffN27d28mT56c8TzNoVhhVq+GW26JUcFz58a0LKecEvuz9Q1ctgy+9a14DB3a2u9ARESqnIKiSKVpbIR7742BHu+9F6tr3HtvzN0nIiJSRgqKIqXgHk3G9fVRy7dsWaxTnFz3t2vXdQd0uMP48fDTn8YI5REjYqm4Qw/V4A8REWkVCooihVq1KkYTp67g8f77TU2/y5ZFSGxujtJkaEwGxx494nlTpsTE03feGc3H7duX7W2JSGaNjfFfvlev/FZrbIn6epg3L8aJ1dTE35LdusWjpqY4fyu6x1Lg48fHokW9e8Omm8bUoZtu2vR1t27r/1qythUrYun6J5+MiSWSC0MNGVL5izYpKJaYu2NtvDaoTU/avnJl/FadMqUpFE6f3rSiR7t20Rdwq62aagqTv92TX3ftGo/GxhiMsnRp5i3AddfBqae2bGUSafOSy1xvtNGGW8nc0BCBat68WIp73rz4EG5Ohw6xAuOOO+Yf9D7+GP75z/jv/89/xuxQEEEqdQXIHXeEbbbJPFd8prImt+lfN9dF3GztXymbbRavnyzDiBFxLJPFiyOcjB8fj+REFz16RNfmTL++u3Rpeo0xY+IxZEh+9y2Ve3SproRfZ59/Xt6JGdzj4yJ535PT13boEB8FjY1xXpcuTUvRJx8jRlTWmEGtzFIkmVZm+eCDD6itraVXr15tNiy6O4sWLaKuro7Bgwe3dnGKwx0mT45avb/8JX7TtmsXvymTy7kNHx4TRm+zTfy5L0BUtiY/+Dp3jlvURn/08/bRRzBpUjzS5x5P19gY4SJZMZ3ac6G+Pv5ugfiwSf9bJPn3Sbdu0L9//LgmHwMHlnfRm5ZavTru14wZ8XjvvQg2qYGqrm79XiNb0Fu9Gp59tumD/c034/y+fSMo7bdf/E2XXAly6tSmaUU32ih+1gcNin/jZHmzlXWTTdatxUtuN9kkrpvp3z/59axZUY7PPovrmcXfq8n3NGwYvPFGvI8XX4yfq+7d4YADmoLflltGeEotb+p9/vjj+DWYDJZbbtn03P33j6CZqr4+XjO5Qubrr8ejrq7p5zX9Z7Vr15iwYejQpn+LIUOK15Dy/vvw4IOx3PvkyfF6yXueqSa1pib7Pa+vj0enTpnrA5JfL13a9AfG3LlRjm23bbp3++wT72/q1HVXFU2d1OLZZ0vfLT3flVkUFIskU1BcvXo1s2fPbvMTWtfU1NC/f386VsMnUXNmzIC77oqAOGNGJJ2jj4YTT4SvfKXNB8Jly+LD8fXXo/mrOWvWRA1Lek3J4sVrnzdkCBx1VNzG3XcvfhPLihUwZ07mGprk159+GrMDDRkSH0ipAarYf7WvXh2/8CdNgueei+3s2XGsa9fcsxSZRQ1Dtg+hbt3iHiY/tJIfYqkfap99FkEi9ddO+/Yxx3ryfW+xRe4P465d1w4wm24a96ulwX/NmrVDybx58UH6wQdNwfDDD9defrumJkJuv37rBqrUD/xstWlJydUaUz+Y33pr7aBnFkG8U6eYkz75wT5iROaf29Wrozk69ZqzZ8fiRZlCYPLRt29xatjcYebMdcPGjBlxvF27+D+XfB+77VZ4jZp7hPXx4+Hxx2HChKbu1rvvDnvsEf9myddNxonu3ZtC62abrf3zmr5dujR+BpL/7skattQgv912sQR7rp+9ZC3eAw/E47XXYv/IkXDQQfHvm/57YtGi5q9p1vR/r0uXphC/bFkE7Uw22QS++tW47wceGP/38rnXs2c3hezTTmv5lLf5UlAss0xBUVrZ1Knwq1/Fb6Dk/H+pK4ckv54xI8Lhc8/Fb4X99oOTTop0U1vb2u+i6Nybfrmnfsik/qLPR21t5g/v5HbevPhr/okn4kO1X7+m0LjPPoXVcLlHqEgv8zvvrB0sIP4Jk32vNt00PmDmzIkPvGTzYVLv3hGcirG88+LFUXvT0BDfb7FFDFjfc8/Y7rBD+Zq+GhvjfiUDWPoj1x8C2dTUNP0b9+mT+/0k+93Nmxf3PtnclmqTTdYO76mPfv1K138rGfSSP0vu8eG+994RCKpVXV28ryFD4me/mFatguefb6p1ffnlqGlMhrrkduDAwv4/pS67nvpI/cOzU6fstYF9+sT4vwceiP/nZvH/7uij43dOc41dq1fHGgLJLgy1tWv/gda5c/b3khoak8G3Y8cIutXQvVxBscwUFCvIqlXw//4f/OIX8b+8V6+m+QazLUG33XZRc3jCCfEJ3wZNnx7zb//5z01NHGbxgZL8yz21FqA57drlXyuydCn8/e/xS/yxxyJI9ewJhx8eHyjNqatras5KbbYdMKCpzEOHrv3B0VyAqavLHJzSA2RLdOkStTZ77hmPSv4xWrmy+T8KkovwZOtP98kn2YNfqs6dMzfxpW7b4N9iG4zGxtIFeff4Ay/5B2Gmvp4LFjT9odihQzSJH300HHlk7t9hoqBYdgqKFeKFF+D734821G99C37720gOSStXxidg6oTV3bvHn4BttDPd5Mnwm99E7V7HjrFQy+jREbK23768IxwbGqIm4oEHYuafTAvNpKqpie6g6R29i11TIiLVp7Exmo7nzYPNN9fvhUIpKJaZgmIrq6+PCap/+9tor/r976PKqgyeey46Hh96aNTIFduiRU1NrnV1Tf13Bg/Onm0bG+Hhh+HXv47y9ewJp58O//Vf+ktbRETyD4qaHkeq35NPxtJ2H3wQPYCvvHLdIXlFtmYN/O1vEcSSKyL+9KcR4E46CY4/PvJqIT7/PJqHU0cNvvZaNL9kUlu7bt+grbaK2rqrrormmoEDIzt/73uaG01ERAqnoCiVqbExZoRdsyY6wZjFNvWxZk0MVrnllkhIEyfG6OQSWr4c7rgjgtj06VGrd911UXn5yCNx7Lzz4Mc/jtFuJ50EX//6up3jFy9ed7Ti1KlNI1U7dowpFfbff+0w2K1b08jk5PPuuqtpmoyknXeGe+6BY48t79xhIiLStqjpuUjU9FxEc+dGwnriidzntm8fyeySS6LnfIksWgTXXx+hcMECGDUKfvSj6DidHsTeeScGUd91V4wu7tYtAtvmmzeFu1mzms7v02ftPnjJedDyHSySOop52rQYULHffm22y6WIiBSB+iiWmYJikTzyCJx8cvQ5vOKKmHshOY19pscuu8SIjBZasCDCVfrkqqnfL1gAf/1rDMQ49NAIiF/5Su4g1tgYfRfvvBPuvTeeP2zYuqFw000V6kREpLwUFMtMQXE9rVgR7bXXXRfp6Z57ou21GclpEQqZr2rVqpgE+fHHm+YBy6Zdu6a5tA48MCouW5pJV62K4NjG5+wWEZEqocEsUj2mTYNvfjM63p11FvzylxkT1dKlMdlrctWLf/87AtigQZkn691yy5ik9Z13miaInTgxago7dIAvfzkqLXffPWbISV8Bo1On4tX0VcJapyIiIoVSUJSi+/zzdZdyy8g92mUvuihGe9w9Pqru6oC6CIaTJzcFwzffjKe0axcDO77znQh0yTVh//WvdddW3Xjjprn6hg6F7343llXad9/KWnRdRESkEikoSovU18eC66mrW7z3Xub1WrMz4KR4rABOyHxW9+5R+3fssbHixe67Z17NwT1W70gt05w5MQL4wAOjhlFERETyp6AoeUuO/P3DH+Cjj9Y+1rNnNPfuumu0Ivfr10yz7VtvwZ/+FHO6HHEEHLA/2LrrQNXUxPWGD8+vH6JZjCDu0ycWqxcREZH1o6AoOX3wQawRfOutMXL34IPhP/9z7f6AeS2dtGQJnHtuXGjbbWH87ZEERUREpCIpKEpWL70UK4/cd1/U6J1wQoz8HT68BRd79FE49dSYI/GnP4WLL9YQYBERkQqnoChrcYd//CMC4oQJ0T/wvPNiMPLmm7fggosXwznnwO23R8J88EHVIoqIiFQJBUX5gnusNPLXv0Yo/PWvoxKwxaODH3442qjnz4cLL4Sf/SzmnBEREZGqoKAoX7jnngiJl1wCF1ywHnP/LVwI//3fMfXN9ttHYNxll6KWVUREREpv3aGmskH69NNoId5tt6j4a1FIXLgw+h8OHhyp82c/i46OCokiIiJVSTWKAsD558f0N48/XtiSeEAshvyb38DvfhfDoo87LgarbLddScoqIiIi5aGgKDz3HNx8cwxaGTmygCfOmxcB8frrYfnymEDxoosUEEVERNoIBcUN3KpVMd5kwAC49NI8n/TJJzHS5fe/h5Ur4VvfioA4bFgpiyoiIiJlpqC4gbvqKpg6NcabdO2axxNeeAH23z9qEE84IQLi1luXvJwiIiJSfgqKG7AZM+Dyy+GYY+Cww/J4wvz5cXKfPjB+PGy1VcnLKCIiIq1HQXED5Q5nnAEdO8Jvf5vHEz7/PPogLlwYnRoVEkVERNo8BcUN1F/+EpWC112X54orF14YS7X88Y+w884lL5+IiIi0Ps2juAFavBjOPjtW0jv99DyeMG4c/OpXcNpp8N3vlrp4IiIiUiFUo7gB+ulPY+rDxx7LY87Et9+OcLj77nDNNeUonoiIiFQI1ShuYCZNghtvjBrFnXbKcXJdHRx1FHTuDPffr3WaRURENjCqUdyArF4dcyZusQVcdlmOk93h5JPh3XfhiSegf/+ylFFEREQqh4LiBuSyy+DNN+Ghh6BbtxwnX3VV9E389a9hv/3KUj4RERGpLGp63kCMHw//8z/wve/BEUfkOHnCBPjJT+DYY+Hcc8tSPhEREak8CoobgDlzYhGV4cNjOpxmzZoF//EfsM02cOutYFaWMoqIiEjlUVBs4z7/PJZiXr4c7rsPunRp5uQPP4RDD4UVK+CBB6C2tmzlFBERkcqjPopt3CWXwLPPwl13wbBhzZz4/PNw5JGwciU8+GCOk0VERGRDoBrFNuwf/4h+iWPHRtNzVvfcA/vuGzWIkyfDAQeUq4giIiJSwRQU26jZs+HEE2HECLj22iwnuUeV4/HHx4Ta//43bLttWcspIiIilUtNz21Qer/Ezp0znLR8eay4cu+9MV/iDTfARhuVu6giIiJSwRQU26Cf/Qz+9S+4++4YvLyOuXPh61+HF1+MNZzPO0+jm0VERGQdCoptzKOPwpVXwqmnRovyOl59FQ4/HBYvjkErRx5Z9jKKiIhIdVBQbENmzYKTToIdd4RrrslwwiuvwN57Q8+eUeU4cmTZyygiIiLVo2IGs5hZfzO71cw+NrOVZjbTzK4xs54FXucoM3vKzJaY2Qozm2ZmF5tZTZaq2/oxAAAgAElEQVTz25vZCWb2rJl9YmYNZvaumf3RzIYX592Vnjt8+9sxu82992bpl3jDDbF94QWFRBEREcmpIoKimQ0BXgJOBl4A/hd4H/ghMNnMeuV5nSuAB4Bdgb8CvwM+Ay4DnjCzTPHpT8BdwKDEc68D3gO+A7xsZvu3+I2V0VtvwTPPwBVXwNZbZzjh889jEu3DD4d+/cpePhEREak+ldL0fD3QFzjL3b9YZM7MrgbOAX4BnNbcBcxsJ+BCYAmwi7u/n9hvwLXAD4CfAJemPGdX4DhgKrCbuzekHDsZuBW4CHhqvd9hiY0bF+NRvvnNLCdMnAgLF8I3vlHOYomIiEgVa/UaRTPbEhgDzCRqAFNdAtQDJ5pZ1xyXOgow4A/JkAjg7g5cADhwupm1T3nOlontk6khMeGhxLZPnm+lVd1/P4weDZttluWE++6Drl3hkEPKWi4RERGpXq0eFIFk0+54d29MPeDudcBzQBdgjxzXSUak99MPJK6zkKi1HJFyaGqyDBmapQ9LbJ/I8bqtbvp0eOMNOOaYLCekNjtn7LwoIiIisq5KCIrJmf7ezXJ8emKbqeddqoWJ7eD0A2ZWC/ROfPvFIsbu/ibRH3J74G0z+52ZXWlmDwO3AH8mmp4r2rhxsT366CwnJJudjzuuXEUSERGRNqASgmKPxHZpluPJ/RvnuM4jie1YMxuUduznRLM0wFqjqN39v4n+j32AM4h+jIcBrwG3u3t9thc0s1PNbIqZTVmwYEGO4pXOuHGw664wYECWE+69F7p1g4MPLmu5REREpLpVQlDMJRnwvLmT3H0ScCMRBF9PTG9zlZk9TwxkSTYzr/niwuFaom/k5cAWQC2wd+L1HjOzM5t5zZvcfZS7j+rTp3W6Mn74IUyZAscem+UENTuLiIhIC1VCUEzWGPbIcrx72nlZuftpwPeBt4jRzKcBq4CDgDcSp81Pecp3gP8CrnX3K919trsvc/d/AYcDy4ErzaxbAe+nrB54ILZZ+ydOmACLFmm0s4iIiBSsEoLiO4lttj6IWyW22fowrsXdb3X3Pdy9a+Kxj7s/AXw5ccqLKacnB6xMyHCdT4C3gW409aOsOOPGxUosQ4ZkOeG++9TsLCIiIi1SCUExGdLGmNla5UkMQhlN1Ow939IXMLMxwEDgaXefk3KoU2Kbrd04uX9VS1+7lD7+GJ57TqOdRUREpDRaPSi6+wxgPLEySnp/wMuArsAdqYNKzGyYmQ1LOxcz655h3xDgJqJv4vlph59NbP/bzHqkPe80oD/wCdGUXXEefDC2OZudNdpZREREWqBSVmY5A5gEXGtmBwDTgN2B/Ygm5wvTzp+W2Fra/lvMbCCxHOBiYCjR17AjMNbd02slrwdOAHYA3jWzvxEru+xMzO+4BjjT3ddQgcaNg2HDYLvtspyQbHY+6KCylktERETahlavUYQvahVHAbcRAfFcYAix9N6X3X1Rnpd6BFhNDGQ5D9gTGAfs7O63ZXjdZUTT9iXAXOB44GxgW+A+YE93f6Cl76uUFiyAp59upjZx9Wo1O4uIiMh6qZQaRdx9FnBynuem1yQm998O3F7g6y4jpsa5vJDntbaHHoLGxmaC4sSJanYWERGR9VIRNYpSuHHjYPBgGDkyywnJSbbV7CwiIiItpKBYhRYvhiefjNpEy1S3unp1jHQ54gg1O4uIiEiLKShWoYcfjiyYdTUWTbItIiIiRaCgWIXGjYP+/WN954w0ybaIiIgUgYJilamrg8cfh6OPhnaZ/vWSo52POAJqaspePhEREWk7FBSrzKOPwsqVOSbZ/vRTjXYWERGR9aagWGXGjYO+fWH06Cwn3Hsv1NZqtLOIiIisNwXFKrJ8edQoHnUUtG+f4YTkaOfDD1ezs4iIiKw3BcUq8vjjUF/fTLPzU0+p2VlERESKRkGxitx/P/TsCfvum+WE++5Ts7OIiIgUjYJilVi5MuZPPPJI6Ngxy0kPPwyHHaZmZxERESkKBcUq8eST8NlnzTQ7NzbC/Pmw1VZlLZeIiIi0XQqKVeKBB6JV+cADs5ywfHlsu3YtW5lERESkbVNQrBK//jU88gh06pTlhIaG2CooioiISJEoKFaJnj1hn32aOaG+PrZdupSlPCIiItL2KSi2FcmgqBpFERERKRIFxbZCTc8iIiJSZAqKbYVqFEVERKTIFBTbCvVRFBERkSJTUGwrVKMoIiIiRaag2Faoj6KIiIgUmYJiW6EaRRERESkyBcW2Qn0URUREpMgUFNuKZFDs3Ll1yyEiIiJtRt5B0cx2KmVBZD01NERtYjtlfxERESmOQlLFS2b2bzP7npmpfbPS1Ner2VlERESKqpCg+CiwM3Az8LGZXWdmI0pTLClYfb0GsoiIiEhR5R0U3f0wYBBwBfAZcCbwqpk9Z2YnmVlNaYooeWloUFAUERGRoiqoQ5u7z3H3S4nAeCTwGLAb8Edgjpn9r5ltW+xCSh5UoygiIiJF1qKRD+7e6O4Pp9QyXg6sAs4C3jSziWZ2bPGKKTmpj6KIiIgUWTGGyA4HdgB6AQYsAvYG/mJmL5nZoCK8huSiGkUREREpshYFRTPra2bnm9kMovn568BE4GhgM2AocCMwEri+OEWVZqmPooiIiBRZh0JONrMDgP8k+id2BBYD1wC/d/f3Uk79ADjDzDoBxxWprNIcNT2LiIhIkeUdFM1sOrAl0bw8hagp/LO7r2jmadMBVXOVg5qeRUREpMgKqVHcHLgNuN7dX8rzOXcDkwstlLSAgqKIiIgUWSFB8UvuvqSQi7v7LGBWYUWSgjU2wooVCooiIiJSVIVMuF1QSJQyamiIrfooioiISBHlHRTN7DQzm2FmX8pyfPPE8e8Xr3iSl/r62KpGUURERIqokOlxjgfmuvvHmQ66+xxgNvDtYhRMCpCsUVRQFBERkSIqJChuA7yW45zXgWEtL460iGoURUREpAQKCYo9gFz9FD8Dera8ONIiyaCoPooiIiJSRIUExbnEUn3N2QFY0PLiSIuoRlFERERKoJCgOAE42Mz2ynTQzPYGDgGeLEbBpADqoygiIiIlUEhQ/CWwCnjCzK42szFmNjyx/V/gn8DKxHlSTmp6FhERkRLIe8Jtd3/HzI4D/gScDfww5bAR/ROPd/dpxS2i5KSmZxERESmBQlZmwd3/bmZbAt8Fdgc2Jga4PA/c7u6Lil5CyU1BUUREREqgoKAIkAiDV5WgLNJS6qMoIiIiJVBIH0WpVPX1YAY1Na1dEhEREWlDCq5RBDCz/sDmQKdMx939mfUplBSovj4Gspi1dklERESkDSkoKJrZGOB/yb36SvsWl0gK19CgZmcREREpurybns1sd+ARYgDL/xEjnZ8BbgbeTnz/MHB58YspzUrWKIqIiIgUUSF9FC8AVgC7untyapwJ7n4asD1wBfBV4P7iFlFyqq9XjaKIiIgUXSFB8cvA39z94/Tne7gEmAZcVsTyST4UFEVERKQECgmKPYCPUr5fBaSnk+eAfda3UFIg9VEUERGREigkKM4HeqZ9PyTtnI5A5/UtlBRIfRRFRESkBAoJiu+ydjB8HjjQzLYGMLPNgGOA6cUrnuRFTc8iIiJSAoUExX8AXzGzTRLf/5aoPXzFzF4kRj73Aa4pbhElJwVFERERKYFCguKNRP/D1QDu/hzwDeADYtTzXOB0d7+j2IWUHNRHUUREREog7wm33f0z4N9p+x4EHix2oaRA6qMoIiIiJVDIhNu3mtk5pSyMtMCaNbBypWoURUREpOgKaXo+HuhbqoJICzU0xFZBUURERIqskKA4EwXFylNfH1s1PYuIiEiRFRIU/wQcYmY9c57ZAmbWP9G8/bGZrTSzmWZ2TaGvZ2ZHmdlTZrbEzFaY2TQzu9jMajKce6mZeY7HjOK9yxJIBkXVKIqIiEiR5T2YBfh/wChggpldBLzo7vOKUQgzGwJMImosHyKm2tkN+CFwsJmNdvdFeVznCuAiYBkwDlgE7EUsKzjGzA509+UpT5nYzOUOB3YGHiv4DZWTgqKIiIiUSCFBcUVia0SYw8wynefuXsh1Aa4nQuJZ7n5dcqeZXQ2cA/wCOK25C5jZTsCFwBJgF3d/P7HfgGuBHwA/AS5NKehEMoRFM2sPfD/x7U0FvpfyUh9FERERKZFCAt2zgBe7AGa2JTCG6AP5u7TDlwCnAiea2bnuXt/MpY4iQuwfkiERIrWa2QXAmcDpZnaFu6/JUaxDgf7A8+7+ekFvqNzUR1FERERKpJB5FPctURn2T2zHu3tj2mvWmdlzRJDcA3iymetslti+n34gcZ2FRK3lCODVHGU6NbGt7NpEUNOziIiIlEwhg1lKZZvE9t0sx5NrR2+d4zoLE9vB6QfMrBbonfh2WHMXMbPNgUOApcBfcrxm61NQFBERkRKphKDYI7FdmuV4cv/GOa7zSGI71swGpR37OdEsDZBrFPVYoD1wl7s35Di39SX7KKrpWURERIos76ZnM7s4z1Pd3a9oYXkyvnTyujledJKZ3Qj8J/C6mY0DPgVGA7sCU4HhQNb+iWbWDvhe4tuczc5mdiqJZuoBAwbkOr00VKMoIiIiJVLIYJZLmzmWDHGW+LqQoJisMeyR5Xj3tPOyF8L9NDN7gQhvxyV2vwQcRIxiHg7Mb+YShwADyHMQi7vfRCJQjho1qugDffKioCgiIiIlUkhQ3C/L/o2JGruzgL8DNxRYhncS22x9ELdKbLP1YVyLu98K3Jq+38z+kPjyxWaenhzEcmM+r1URGhqgXTvo1Km1SyIiIiJtTCGjnp9u5vBDZvYX4AXgzwWWYUJiO8bM2qWOfE4MQhkNLAeeL/C6XzCzMcBA4Gl3n5PlnC8BXyNqLu9t6WuVXX199E/MPKeliIiISIsVbTCLu79BTMR9QYHPmwGMBwYRcx2mugzoCtyROoeimQ0zs3VGL5tZ9wz7hhDNw2uA85spyveJQSx3VsUglqT6ejU7i4iISEkUuoJKLh8RS98V6gxiCb9rzewAYBqwO9Hc/S6x4kqqaYltejXaLWY2kOiXuBgYmihPR2Csu2eslUwMYqmOlVjSKSiKiIhIiRR7epzdiWbigiRqFUcBtyWucS4whFh678v5rPOc8AiwmhjIch6wJ7Hm887uflszzzuIaJp+PlEzWj0aGhQURUREpCQKmR4n2/wvHYAtgFOAvWhh/z53nwWcnOe52RaZvh24vQWv/Rjr1k5Wh2QfRREREZEiK6TpeSbNz2VoxCoq561PgaRAanoWERGREikkKN5B5qDYSPQHfAF4yN1XFqNgkqf6ethkk9YuhYiIiLRBhUyP890SlkNaqqFBTc8iIiJSEpWw1rOsDzU9i4iISInkHRTNbIiZnWRmvbIc7504vmXxiic5KSiKiIhIiRRSo3g+cBXwWZbjS4HfAD9a30JJATQ9joiIiJRIIUFxX+AJd1+d6WBi/z+B/YtQLsnH55/DqlXqoygiIiIlUUhQ3JyYIqc5HwFfanFppDD1iVUNVaMoIiIiJVBIUFwFrLOWcppamp9rUYpJQVFERERKqJCg+CbwNTPrmOmgmW0EHAa8VYyCSR4aGmKrpmcREREpgUKC4l3AAOBeM9ss9UDi+3uJpfzuKF7xpFmqURQREZESKmRllpuAY4AjgQPN7HVgDtF3cQegC/AEcEOxCylZKCiKiIhICeVdo+jujcChwJXAamAPIjjuQfRf/B/ga4nzpBwUFEVERKSECqlRTE6Bc4GZXQQMAzYGlgBvKyC2AvVRFBERkRIqKCgmJUKhBq20NtUoioiISAlpCb9qpqAoIiIiJaQl/KpZsulZQVFERERKQEv4VbNkjaL6KIqIiEgJaAm/alZfD+3bw0YbtXZJREREpA3SEn7VrL4+mp3NWrskIiIi0gZpCb9q1tCgZmcREREpGS3hV82SNYoiIiIiJaAl/KqZgqKIiIiUkJbwq2YKiiIiIlJChTQ94+6r3f0CoBewPbBXYtvb3S8C1pjZkcUvpmSkPooiIiJSQkVZws/MBprZWOBkoB/QvjjFk2bV10Pv3q1dChEREWmjWhQUAcysPdFf8VTgq0TtpBP9FKUc1PQsIiIiJVRwUEys5TwW+C6waWL3QuBG4BZ3/7BopZPmqelZRERESiivoGhmHYCjiNrD/Yjaw1XAA8SAlofc/eJSFVKyUI2iiIiIlFCzQdHMtgJOAb4D9AYMeBm4DfiTu39qZhrl3FoUFEVERKSEctUovkP0O5wP/C/wR3efWvJSSW6rVsHnnysoioiISMnkMz2OA48C9yskVpCGhtiqj6KIiIiUSK6g+DPgQ2Lam+fM7C0z+7GZ9St90aRZ9fWxVY2iiIiIlEizQdHdf+HuQ4BDgAeBIcTKLB+Z2d/N7LgylFEyUVAUERGREstrZRZ3f9zdjwW2AC4gahkPAe4hmqZHmtkuJSulrEtBUUREREqs0CX85rv7le4+FDgQuJ9Y93kU8IKZvWJmZ5agnJJOfRRFRESkxAoKiqnc/Ul3/w+gP/Bj4F1gR+DaIpVNmqMaRRERESmxFgfFJHdf6O6/cfdtgf2J5mgpNQVFERERKbEWr/WcibtPBCYW85qShZqeRUREpMTWu0ZRWolqFEVERKTEFBSrlYKiiIiIlJiCYrVSUBQREZESU1CsVg0N0KEDdOzY2iURERGRNkpBsVrV16s2UUREREpKQbFaKSiKiIhIiSkoVquGBk2NIyIiIiWloFitVKMoIiIiJaagWK0UFEVERKTEFBSrlYKiiIiIlJiCYrVSH0UREREpMQXFaqUaRRERESkxBcVqpaAoIiIiJaagWK0UFEVERKTEFBSrkbv6KIqIiEjJKShWo1WrYM0a1SiKiIhISSkoVqP6+tgqKIqIiEgJKShWo4aG2KrpWUREREpIQbEaqUZRREREykBBsRopKIqIiEgZKChWIwVFERERKQMFxWqkPooiIiJSBgqK1Ug1iiIiIlIGFRMUzay/md1qZh+b2Uozm2lm15hZzwKvc5SZPWVmS8xshZlNM7OLzawmx/OOMLPHzGxB4vVnmdnfzGyP9XtnJaCgKCIiImXQobULAGBmQ4BJQF/gIeBtYDfgh8DBZjba3RflcZ0rgIuAZcA4YBGwF3AZMMbMDnT35WnPaQfcAJwCzAIeSDxvU2APYBfg+SK8zeJRUBQREZEyqIigCFxPhMSz3P265E4zuxo4B/gFcFpzFzCznYALgSXALu7+fmK/AdcCPwB+Alya9tRziZB4JzDW3VelXbdji99VqaiPooiIiJRBqzc9m9mWwBhgJvC7tMOXAPXAiWaWq/rsKMCAPyRDIoC7O3AB4MDpZtY+5bW7AxcDs4FT0kNi4vmrC31PJacaRRERESmDVg+KwP6J7Xh3b0w94O51wHNAF6IZuDmbJbbvpx9IXGchUWs5IuXQEUA34M9AOzM71szON7MzzWzHgt9JudTXw0YbQYdKqRAWERGRtqgSksY2ie27WY5PJ2octwaebOY6CxPbwekHzKwW6J34dhjwauLrXRPb1cA0YGDa88YBJ7l7QzOvW34NDWp2FhERkZKrhBrFHont0izHk/s3znGdRxLbsWY2KO3Yz4lmaYDUUdR9E9sfAwuA3YHaxHYKcAzRf7Ky1Ner2VlERERKrhKCYi7JgOfNneTuk4AbiSD4upn90cyuMrPniYEsUxOnrkl5WrK/4nLgcHd/wd2XufsLRLP0MqJ/5OYZC2Z2qplNMbMpCxYsaNGbaxEFRRERESmDSgiKyRrDHlmOd087Lyt3Pw34PvAWcBwxUnoVcBDwRuK0+SlPWZzYPu/un6Rday7wb+Iejcryeje5+yh3H9WnT59cxSseBUUREREpg0roo/hOYrt1luNbJbbZ+jCuxd1vBW5N329mf0h8+WKG116S5XLJINk5n9cuG/VRFBERkTKohBrFCYntmMTk119IDEIZTTQNt3jSazMbQwxUedrd56QcSg6OGZ7lqcn9M1v62iWhGkUREREpg1YPiu4+AxgPDALOTDt8GdAVuMPd65M7zWyYmQ1Lv1ZiXsT0fUOAm4i+ieenvfZrxPQ725rZ2LTnjQW2BWawdi1k61NQFBERkTKohKZngDOIJfyuNbMDiKlqdgf2I5qcL0w7f1pia2n7bzGzgcBLRLPxUOBwoCOx6kqmWsnvA/8Cbjazo4lBL9sBhwINwHfdfU2G57We+no1PYuIiEjJtXqNInxRqzgKuI0IiOcCQ4il976czzrPCY8QcyIeB5wH7Ems+byzu9+W5bXfAXYGbgF2JNaX3gW4Bxjl7v9q0ZsqpYYG1SiKiIhIyVVKjSLuPgs4Oc9z02sSk/tvB25v4WuPzXlipVDTs4iIiJRBRdQoSgHcFRRFRESkLBQUq83KlREW1UdRRERESkxBsdrUJwZ/q0ZRRERESkxBsdooKIqIiEiZKChWGwVFERERKRMFxWrT0BBb9VEUERGRElNQrDaqURQREZEyUVCsNgqKIiIiUiYKitUmGRTV9CwiIiIlpqBYbZJ9FFWjKCIiIiWmoFht1PQsIiIiZaKgWG0UFEVERKRMFBSrjabHERERkTJRUKw29fXQqRO0b9/aJREREZE2TkGx2tTXq9lZREREykJBsdrU16vZWURERMpCQbHaNDSoRlFERETKQkGx2qjpWURERMpEQbHaKCiKiIhImSgoVhv1URQREZEyUVCsNuqjKCIiImWioFht1PQsIiIiZaKgWG0UFEVERKRMFBSrTUOD+iiKiIhIWSgoVhN39VEUERGRslFQrCbLl0dYVFAUERGRMlBQrCb19bFV07OIiIiUgYJiNWloiK1qFEVERKQMFBSrSbJGUUFRREREykBBsZooKIqIiEgZKShWE/VRFBERkTJSUKwm6qMoIiIiZaSgWE3U9CwiIiJlpKBYTdT0LCIiImWkoFhN1PQsIiIiZaSgWE3U9CwiIiJlpKBYTdT0LCIiImWkoFhN6uuhpgba6Z9NRERESk+Jo5o0NKjZWURERMpGQbGa1NcrKIqIiEjZKChWEwVFERERKSMFxWpSX6+BLCIiIlI2CorVRH0URUREpIwUFKuJmp5FRESkjBQUq4mankVERKSMFBSriZqeRUREpIwUFKuJmp5FRESkjBQUq4mCooiIiJSRgmK1aGyE5cvVR1FERETKRkGxWixfHlvVKIqIiEiZKChWi/r62CooioiISJkoKFaLZFBU07OIiIiUiYJitVCNooiIiJSZgmK1aGiIrYKiiIiIlImCYrVQjaKIiIiUmYJitVAfRRERESkzBcVqoaZnERERKTMFxWqhpmcREREpMwXFaqGgKCIiImWmoFgt1EdRREREykxBsVok+yh27ty65RAREZENhoJitaivj9rEdvonExERkfJQ6qgWQ4bAQQe1dilERERkA1IxQdHM+pvZrWb2sZmtNLOZZnaNmfUs8DpHmdlTZrbEzFaY2TQzu9jMarKc7808ni/OuyuC00+HBx5o7VKIiIjIBqRDaxcAwMyGAJOAvsBDwNvAbsAPgYPNbLS7L8rjOlcAFwHLgHHAImAv4DJgjJkd6O7LMzz1Q+C2DPtnF/5uRERERNqGigiKwPVESDzL3a9L7jSzq4FzgF8ApzV3ATPbCbgQWALs4u7vJ/YbcC3wA+AnwKUZnj7T3TPtFxEREdlgtXrTs5ltCYwBZgK/Szt8CVAPnGhmuSYQPAow4A/JkAjg7g5cADhwupm1L1LRRURERNq0Vg+KwP6J7Xh3b0w94O51wHNAF2CPHNfZLLF9P/1A4joLiVrLERmeu7GZfc/MLjCzM80s12uJiIiItHmV0PS8TWL7bpbj04kax62BJ5u5zsLEdnD6ATOrBXonvh0GvJp2yo7ALWnPeQ040d3faOY1RURERNqsSqhR7JHYLs1yPLl/4xzXeSSxHWtmg9KO/ZxolgZIH0V9NTAa6APUArsC9xPh8Skz2zzbC5rZqWY2xcymLFiwIEfxRERERKpLJQTFXJIBz5s7yd0nATcSQfB1M/ujmV2VmOLmB8DUxKlr0p53rrtPcveF7r7M3ae4+zeIUdO9gfOaec2b3H2Uu4/q06dPy96diIiISIWqhKCYrDHskeV497TzsnL304DvA28BxxEjpVcBBwHJJuT5eZbrhsR2nzzPFxEREWlTKqGP4juJ7dZZjm+V2Gbrw7gWd78VuDV9v5n9IfHli3mWK9mWnGu0tYiIiEibVAk1ihMS2zFmtlZ5EoNQRgPLgRavkmJmY4CBwNPuPifPpyVHPq8zilpERERkQ9DqQdHdZwDjgUHAmWmHLyNq9O5w9/rkTjMbZmbD0q9lZt0z7BsC3ET0TTw/7djOmeZnNLMdiEm+Ae4q5P2IiIiItBWV0PQMcAaxhN+1ZnYAMA3YHdiPaHK+MO38aYmtpe2/xcwGAi8Bi4GhwOFAR2Csu6fXSp4FHG1mTwGzgJXE9DkHA+2Bm4F71vvdiYiIiFShigiK7j7DzEYBlxMh7VBgLrH03mXu/mmel3oEOJUYyFJLDFwZB/zK3V/PcP5ficEyOxATf9cQ60M/Btzs7n9r8ZsSERERqXIWK9zJ+jKzBcCHJX6Z3jRNLC7Fo/taGrqvpaH7Whq6r6Wh+1oaxbivA90959x+CopVxMymuPuo1i5HW6P7Whq6r6Wh+1oauq+loftaGuW8r60+mEVEREREKpOCooiIiIhkpKBYXW5q7QK0UbqvpaH7Whq6r6Wh+1oauq+lUbb7qj6KIiIiIpKRahRFREREJCMFRRERERHJSEGxwplZfzO71cw+NrOVZjbTzK4xs56tXbZKZ2bHmtl1ZvasmX1mZm5mzS7JaGZ7mtmjZvapmTWY2etmdraZtS9XuSuZmfUys7Fm9qCZvWdmy81sqZn9y8y+n75ee8rzdF9zMLNfmtmTZjYrcV8/NbNXzOwSM+uV5Tm6rwUysxMTvwvczEQlXXwAAAtsSURBVMZmOecwM5uY+NleZmb/NrPvlLuslSzxWeRZHp9keY5+XvNkZnub2Tgzm5v47J9rZuPN7NAM55b0vqqPYgVLrFM9CegLPAS8DexGLG34DjDa3Re1Xgkrm5m9CuwILANmE8sz3u3u385y/pHESj4rgL8AnxJLQG4D3O/u3yhHuSuZmZ0G/J5YOWkC8BGwKXA00IO4f9/wlF8suq/5MbNVwMvAW8SqUl2BPYBRwMfAHu4+K+V83dcCmdkWwBvEEq3dgFPc/Q9p5/wAuI5YpesvwCrgWKA/cJW7n1fWQlcoM5sJbAxck+HwMnf/Tdr5+nnNk5ldBFxBTKj9CPH7tjewEzDB3X+ccm7p76u761GhD+BxwIH/Stt/dWL/Da1dxkp+EIF6K2JN8H0T9+yuLOd2Jz6cVwKjUvbXEGHdgW+29ntq7Qex1OXhQLu0/ZsRodGBY3RfW3Rva7Ls/0XiPl2v+7pe99eAJ4AZwK8T92hs2jmDiA/cRcCglP09gfcSz/lya7+XSngAM4GZeZ6rn9f87+s3Evfjn0BthuMdy31f1fRcocxsS2AM8Z/xd2mHLwHqgRPNrGuZi1Y13H2Cu0/3xP+cHI4F+gB/dvcpKddYAVyU+Pb0EhSzqrj7U+7+sLs3pu3/BLgh8e2+KYd0X/OUuCeZ3JvYbpWyT/e1cGcRf+icTPz+zOR7QCfg/9x9ZnKnuy8G/ifx7WklLGNbpZ/XPCS67vwSaACOd/e69HPcfXXKt2W5rx3W9wJSMvsntuMzfCjXmdlzRJDcA3iy3IVrg5L3+x8Zjj1D/Mfd08w6ufvK8hWrqiR/gX2esk/3df0dnti+nrJP97UAZrYtcCXwW3d/xsz2z3Jqc/f1sbRzBDqZ2beBAUT4fh14xt3XpJ2nn9f87AkMBu4HFpvZ14DtiVruF9x9ctr5ZbmvCoqVa5vE9t0sx6cTQXFrFBSLIev9dvfPzewDYDiwJTCtnAWrBmbWATgp8W3qLy3d1wKZ2XlE/7keRP/EvYgP4CtTTtN9zVPiZ/NOomvEBTlOb+6+zjWzeqC/mXVx94bilrQqbUbc21QfmNnJ7v50yj79vOZn18R2HtFfeUTqQTN7BjjW3RckdpXlvqrpuXL1SGyXZjme3L9xGcqyIdD9Xj9XEn/5Puruj6fs130t3HlE95KziZD4D2BMyocD6L4W4mJiEMB33X15jnPzva89shzfkPwROIAIi12JUHMj0c/zMTPbMeVc/bzmp29iexrQGfgqUEv8bn0c2Ae4L+X8stxXBcXqZYmthq2Xh+53FmZ2FnAuMSr/xEKfntjqvia4+2bubsQH8NFEbcArZrZzAZfRfQXMbDeiFvGqDM12LbpkYrtB31cAd78s0Wd5nrs3uPub7n4aMdiyM3BpAZfTfQ3/v717j5WjqgM4/v1ZU8QIrfigYIgSqRYhCiZYpU0fQbHEFNoEbXy1MfgH0ZhAopEQCNeoCYkxjY8YE5U0DdKCJvjWRK0UWh/Y2CoKtiq0QR62PFpAaRvk5x/nrI6b2bu77X219/tJJid75szcM7+79+5vZ+ac6UxnE5Qzhz/PzGcy80/ASsrsHYsj4m0D7m9M4mqiOHX1++Z6clc7HR3jfQQi4qPAFyhTuizNzCe6mhjXI1Q/gG+n3GLyMmB9Y7Vx7aNxyXkXcP2Amw0a16eOomvHu86gtkWNOt+vg3mylvdn5u+bK+rZ8M7VmrfUckLiaqI4de2s5et6rO+MgOx1D6OG0zPe9QPnTMogjfsnslNTWURcBXwZ+CMlSWybZNe4HqXM3ENJxM+JiJfXauPa30so8TkbONicEJpyaR/ga7WuMxfgaHE9jXKJ9e/enziqvbVszsjh+3UwnTjt77G+k0ie2NV+XONqojh1/aKWF3c/7SIiTgIWAM8Cv57ojh2nNtVyWcu6RcCLgV9O8xF5/xURnwTWAjsoSeLeHk2N69g4vZad0aTGtb9DwDd6LNtrmy31deey9GhxvaSrjdp1Los2kxPfr4O5k5LYzY2ImS3rz63l7lpOTFwne3JJl1En3nTC7bGL5RL6T7i9DyeEHSSW19d4bANO6dPWuA4W03nAnJb6F/C/Cbe3Gtcxi/cI7RNun4kTbg8Sv3Pa/vaBV1Nm5Ejg2ka979fBY3tzjcdnuurfATxPOds4eyLj6iP8prCWR/jdB8ynPHFkF3Bh+gi/niJiBbCivpwDvJPyLfeuWvdYNh7HVdt/m/JBsZHyKKRLqY9CAt6T0/wPpj7vdh3lzNaXaL/3ZXdmrmtsY1z7qJfxP0c5o/A3SqJyKrCYMpjlUeCizLy3sY1xPUIRMUK5/Nz2CL+PAV/ER/j1VON3DeXK1wPA08BrgXdRkpQfASsz83BjG9+vA4iIVwJbgbMon1V3UxLwlZTE732Z+a1G+/GP62Rnzy59v12cQZmG4BHKP6w9lMEDo57Jcfm/swa9lt0t2yyg/JN7knJp/x7gamDGZB/PVFgGiGkCdxjXoeN6LuUJTDsoz3d9jpKE/7bGvPXv3bgecbw77+MP91i/HNhMSYD+WX8Paya731NloXyB2UCZ6WA/ZbL9fZTHzq2GchKqZTvfr4PF9xTKlcMH6uf+45STRW+djLh6RlGSJEmtHMwiSZKkViaKkiRJamWiKEmSpFYmipIkSWploihJkqRWJoqSJElqZaIoSZKkViaKkjRNRMRIRGRELJnsvkg6NpgoStKAapLVb1ky2f2UpLHywsnugCQdgz41yrrdE9UJSRpvJoqSNKTMHJnsPkjSRPDSsySNk+Y9gRGxJiK2R8SzEbE3Im6KiDk9tpsbEesj4qGIOBwRD9fXc3u0nxERV0bE1og4UH/GXyPi66Nsc3lE3B0R/4qIJyJiY0S8aiyPX9KxzzOKkjT+rgYuBm4FfgIsBD4ELImI+Zm5r9MwIi4AfgacBHwPuBeYB7wfuCwiLsrMbY32M4EfAm8HHgRuAZ4CXgOsBLYAf+nqz0eAS+v+NwPzgVXAmyLivMw8NJYHL+nYZaIoSUOKiJEeqw5m5o0t9ZcA8zNze2Mfa4GrgBuBK2pdAOuBk4EPZOY3G+1XARuBmyPiDZn5fF01QkkSvw+8u5nkRcQJdV/dlgEXZOY9jba3AO8FLgNu63nwkqaVyMzJ7oMkHRMiot8/zAOZObvRfgS4AbgpM6/o2tcsYA9wAjA7Mw9FxALKGcBfZeaFLT//LsrZyMWZeWdEzAAeB2YCZ2Xmw3363+nPZzPzuq51S4FNwOcz8+N9jlPSNOE9ipI0pMyMHsvsHptsbtnHAWAH8CLg7Fr95lpu6rGfTv35tZwHzAL+0C9J7LKtpe7BWr50iP1IOs6ZKErS+PtHj/pHazmrq3ykR/tO/eyu8qEh+7O/pe65Ws4Ycl+SjmMmipI0/k7tUd8Z9Xygq2wdDQ2c1tWuk/A5WlnSuDBRlKTxt7i7ot6jeB5wELivVncGuyzpsZ9O/e9q+WdKsvjGiDh9LDoqSU0mipI0/j4YEed31Y1QLjVvaIxU3grsBBZGxOXNxvX1ImAXZcALmflv4CvAicBX6yjn5jYzI+IVY3wskqYRp8eRpCGNMj0OwHcyc0dX3Y+BrRFxG+U+w4V12Q1c02mUmRkRa4CfArdGxHcpZw1fD6wAngZWN6bGgfI4wfnAcmBXRPygtjuDMnfjJ4B1R3SgkqY9E0VJGt4No6zbTRnN3LQWuJ0yb+Iq4BlK8nZtZu5tNszM39RJt6+jzI+4HHgM2AB8OjN3drU/HBHLgCuB1cAaIICH68/cMvzhSVLhPIqSNE4a8xYuzcw7Jrc3kjQ871GUJElSKxNFSZIktTJRlCRJUivvUZQkSVIrzyhKkiSplYmiJEmSWpkoSpIkqZWJoiRJklqZKEqSJKmViaIkSZJa/QddN9sCd2dm1QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(1, 1, figsize=(10,6))\n",
    "ax.plot(np.sqrt(lr_model_history.history['acc']), 'r', label='train')\n",
    "ax.plot(np.sqrt(lr_model_history.history['val_acc']), 'b' ,label='val')\n",
    "ax.set_xlabel(r'Epoch', fontsize=20)\n",
    "ax.set_ylabel(r'Accuracy', fontsize=20)\n",
    "ax.legend()\n",
    "ax.tick_params(labelsize=20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Apply a custon learning rate change using `LearningRateScheduler`\n",
    "Write a function that performs the exponential learning rate decay as indicated by the following formula:\n",
    "\n",
    "\\begin{align}\n",
    "lr = lr0 * e^{(-kt)} \n",
    "\\textrm{}\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# solution\n",
    "epochs = 60\n",
    "learning_rate = 0.1 # initial learning rate\n",
    "decay_rate = 0.1\n",
    "momentum = 0.8\n",
    "\n",
    "# define the optimizer function\n",
    "sgd = SGD(lr=learning_rate, momentum=momentum, decay=decay_rate, nesterov=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_dim = x_train.shape[1]\n",
    "num_classes = 10\n",
    "batch_size = 196\n",
    "\n",
    "# build the model\n",
    "exponential_decay_model = Sequential()\n",
    "exponential_decay_model.add(Dense(64, activation=tf.nn.relu, kernel_initializer='uniform', input_dim = input_dim))\n",
    "exponential_decay_model.add(Dropout(0.1))\n",
    "exponential_decay_model.add(Dense(64, kernel_initializer='uniform', activation=tf.nn.relu))\n",
    "exponential_decay_model.add(Dense(num_classes, kernel_initializer='uniform', activation=tf.nn.softmax))\n",
    "\n",
    "# compile the model\n",
    "exponential_decay_model.compile(loss='categorical_crossentropy', \n",
    "                                optimizer=sgd, \n",
    "                                metrics=['acc'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define the learning rate change \n",
    "def exp_decay(epoch):\n",
    "    lrate = learning_rate * np.exp(-decay_rate*epoch)\n",
    "    return lrate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 60000 samples, validate on 10000 samples\n",
      "Epoch 1/60\n",
      "60000/60000 [==============================] - 1s 16us/step - loss: 1.9924 - acc: 0.3865 - val_loss: 1.4953 - val_acc: 0.5841\n",
      "Epoch 2/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 1.2430 - acc: 0.6362 - val_loss: 1.0153 - val_acc: 0.7164\n",
      "Epoch 3/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.9789 - acc: 0.7141 - val_loss: 0.8601 - val_acc: 0.7617\n",
      "Epoch 4/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.8710 - acc: 0.7452 - val_loss: 0.7811 - val_acc: 0.7865\n",
      "Epoch 5/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.8115 - acc: 0.7609 - val_loss: 0.7336 - val_acc: 0.7968\n",
      "Epoch 6/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.7749 - acc: 0.7678 - val_loss: 0.7030 - val_acc: 0.8035\n",
      "Epoch 7/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.7524 - acc: 0.7742 - val_loss: 0.6822 - val_acc: 0.8095\n",
      "Epoch 8/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.7342 - acc: 0.7788 - val_loss: 0.6673 - val_acc: 0.8122\n",
      "Epoch 9/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.7218 - acc: 0.7840 - val_loss: 0.6562 - val_acc: 0.8148\n",
      "Epoch 10/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.7144 - acc: 0.7836 - val_loss: 0.6475 - val_acc: 0.8168\n",
      "Epoch 11/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.7054 - acc: 0.7857 - val_loss: 0.6408 - val_acc: 0.8175\n",
      "Epoch 12/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.7008 - acc: 0.7896 - val_loss: 0.6354 - val_acc: 0.8185\n",
      "Epoch 13/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.6950 - acc: 0.7885 - val_loss: 0.6311 - val_acc: 0.8197\n",
      "Epoch 14/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.6921 - acc: 0.7895 - val_loss: 0.6274 - val_acc: 0.8199\n",
      "Epoch 15/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.6888 - acc: 0.7913 - val_loss: 0.6244 - val_acc: 0.8204\n",
      "Epoch 16/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.6833 - acc: 0.7932 - val_loss: 0.6219 - val_acc: 0.8206\n",
      "Epoch 17/60\n",
      "60000/60000 [==============================] - 1s 12us/step - loss: 0.6831 - acc: 0.7942 - val_loss: 0.6199 - val_acc: 0.8208\n",
      "Epoch 18/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.6822 - acc: 0.7937 - val_loss: 0.6182 - val_acc: 0.8212\n",
      "Epoch 19/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.6790 - acc: 0.7955 - val_loss: 0.6167 - val_acc: 0.8215\n",
      "Epoch 20/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.6797 - acc: 0.7935 - val_loss: 0.6155 - val_acc: 0.8218\n",
      "Epoch 21/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.6773 - acc: 0.7953 - val_loss: 0.6144 - val_acc: 0.8222\n",
      "Epoch 22/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.6742 - acc: 0.7960 - val_loss: 0.6135 - val_acc: 0.8227\n",
      "Epoch 23/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.6746 - acc: 0.7958 - val_loss: 0.6127 - val_acc: 0.8236\n",
      "Epoch 24/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.6729 - acc: 0.7973 - val_loss: 0.6120 - val_acc: 0.8237\n",
      "Epoch 25/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.6749 - acc: 0.7963 - val_loss: 0.6114 - val_acc: 0.8238\n",
      "Epoch 26/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.6715 - acc: 0.7967 - val_loss: 0.6109 - val_acc: 0.8241\n",
      "Epoch 27/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.6728 - acc: 0.7975 - val_loss: 0.6105 - val_acc: 0.8241\n",
      "Epoch 28/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.6721 - acc: 0.7964 - val_loss: 0.6101 - val_acc: 0.8246\n",
      "Epoch 29/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.6707 - acc: 0.7972 - val_loss: 0.6098 - val_acc: 0.8247\n",
      "Epoch 30/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.6711 - acc: 0.7980 - val_loss: 0.6095 - val_acc: 0.8247\n",
      "Epoch 31/60\n",
      "60000/60000 [==============================] - 1s 12us/step - loss: 0.6721 - acc: 0.7960 - val_loss: 0.6092 - val_acc: 0.8248\n",
      "Epoch 32/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.6708 - acc: 0.7981 - val_loss: 0.6090 - val_acc: 0.8249\n",
      "Epoch 33/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.6720 - acc: 0.7968 - val_loss: 0.6088 - val_acc: 0.8249\n",
      "Epoch 34/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.6700 - acc: 0.7973 - val_loss: 0.6086 - val_acc: 0.8250\n",
      "Epoch 35/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.6714 - acc: 0.7979 - val_loss: 0.6084 - val_acc: 0.8250\n",
      "Epoch 36/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.6687 - acc: 0.7980 - val_loss: 0.6083 - val_acc: 0.8250\n",
      "Epoch 37/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.6700 - acc: 0.7963 - val_loss: 0.6082 - val_acc: 0.8250\n",
      "Epoch 38/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.6702 - acc: 0.7964 - val_loss: 0.6081 - val_acc: 0.8252\n",
      "Epoch 39/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.6711 - acc: 0.7963 - val_loss: 0.6080 - val_acc: 0.8250\n",
      "Epoch 40/60\n",
      "60000/60000 [==============================] - 1s 12us/step - loss: 0.6698 - acc: 0.7971 - val_loss: 0.6079 - val_acc: 0.8251\n",
      "Epoch 41/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.6696 - acc: 0.7967 - val_loss: 0.6079 - val_acc: 0.8252\n",
      "Epoch 42/60\n",
      "60000/60000 [==============================] - 1s 12us/step - loss: 0.6699 - acc: 0.7989 - val_loss: 0.6078 - val_acc: 0.8252\n",
      "Epoch 43/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.6701 - acc: 0.7952 - val_loss: 0.6077 - val_acc: 0.8252\n",
      "Epoch 44/60\n",
      "60000/60000 [==============================] - 1s 12us/step - loss: 0.6704 - acc: 0.7973 - val_loss: 0.6077 - val_acc: 0.8252\n",
      "Epoch 45/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.6697 - acc: 0.7971 - val_loss: 0.6076 - val_acc: 0.8252\n",
      "Epoch 46/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.6718 - acc: 0.7969 - val_loss: 0.6076 - val_acc: 0.8252\n",
      "Epoch 47/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.6708 - acc: 0.7965 - val_loss: 0.6076 - val_acc: 0.8252\n",
      "Epoch 48/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.6690 - acc: 0.7968 - val_loss: 0.6075 - val_acc: 0.8252\n",
      "Epoch 49/60\n",
      "60000/60000 [==============================] - 1s 12us/step - loss: 0.6700 - acc: 0.7959 - val_loss: 0.6075 - val_acc: 0.8252\n",
      "Epoch 50/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.6722 - acc: 0.7963 - val_loss: 0.6075 - val_acc: 0.8252\n",
      "Epoch 51/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.6698 - acc: 0.7950 - val_loss: 0.6075 - val_acc: 0.8252\n",
      "Epoch 52/60\n",
      "60000/60000 [==============================] - 1s 12us/step - loss: 0.6689 - acc: 0.7978 - val_loss: 0.6075 - val_acc: 0.8252\n",
      "Epoch 53/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.6696 - acc: 0.7973 - val_loss: 0.6074 - val_acc: 0.8252\n",
      "Epoch 54/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.6678 - acc: 0.7975 - val_loss: 0.6074 - val_acc: 0.8252\n",
      "Epoch 55/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.6694 - acc: 0.7969 - val_loss: 0.6074 - val_acc: 0.8252\n",
      "Epoch 56/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.6688 - acc: 0.7981 - val_loss: 0.6074 - val_acc: 0.8252\n",
      "Epoch 57/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.6674 - acc: 0.7988 - val_loss: 0.6074 - val_acc: 0.8252\n",
      "Epoch 58/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.6685 - acc: 0.7970 - val_loss: 0.6074 - val_acc: 0.8252\n",
      "Epoch 59/60\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.6702 - acc: 0.7953 - val_loss: 0.6074 - val_acc: 0.8252\n",
      "Epoch 60/60\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "60000/60000 [==============================] - 1s 11us/step - loss: 0.6701 - acc: 0.7965 - val_loss: 0.6074 - val_acc: 0.8252\n"
     ]
    }
   ],
   "source": [
    "# learning schedule callback\n",
    "loss_history = History()\n",
    "lr_rate = LearningRateScheduler(exp_decay)\n",
    "callbacks_list = [loss_history, lr_rate]\n",
    "\n",
    "# you invoke the LearningRateScheduler during the .fit() phase\n",
    "exponential_decay_model_history = exponential_decay_model.fit(x_train, y_train,\n",
    "                                    batch_size=batch_size,\n",
    "                                    epochs=epochs,\n",
    "                                    callbacks=callbacks_list,\n",
    "                                    verbose=1,\n",
    "                                    validation_data=(x_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['val_loss', 'val_acc', 'loss', 'acc', 'lr'])"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check on the variables that can show me the learning rate decay\n",
    "exponential_decay_model_history.history.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAooAAAGJCAYAAAANCBK/AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3XmUFNX5xvHvOyAIiIgsbqAIirjEdWQRgwIRNRHcMT+V4B5JDC6YaIwR0bjFDTUxatySmMR9jxHirojioKJxA0FcABHBICIIOO/vj1stTdM93T3TPdXd83zOqVPTVdW3364zMo+3qu41d0dEREREJFVV3AWIiIiISGlSUBQRERGRtBQURURERCQtBUURERERSUtBUURERETSUlAUERERkbQUFEVEREQkLQVFEREREUlLQVFERERE0lJQFBEREZG0msddQKXo2LGjd+vWLe4yRERERLKaOnXq5+7eKdtxCooF0q1bN2pqauIuQ0RERCQrM/swl+N06VlERERE0lJQFBEREZG0FBRFREREJC0FRRERERFJS0FRRERERNJSUBQRERGRtEomKJpZFzO71czmmtk3ZjbbzMabWfs82tjHzK40syfNbJGZuZm9kMP7tjOzu83sMzNbbmbvmdk4M2vVsG8lIiIiUr5KYhxFM+sBvAh0Bh4C3gV6A6cC+5lZf3dfmENTPwcOBJYD7wNZQ6aZ9QGeAtYB7gU+BgYB5wGDzWywu3+T95cSERERKXOl0qN4PSEkjnb3g9z9bHcfBFwNbANclGM7lwE7AOsBQ7MdbGbNgNuA1sBh7n6ku58F9AHuA/oDp+f7ZUREREQqQexB0cy6A0OA2cAfU3aPBZYCI8ysTba23H2yu7/l7t/m+PF7AdsCz7n7w0nt1AK/il6ebGaWY3siIiIiFSP2oEi4zAswMQpo33H3JcAkQo9f3yJ+9uOpO9x9FjAd2ALoXoTPFhERESlppRAUt4nW0zPsnxGte1bYZ+dn8WK45x74NtfOUhEREZGGKYWg2C5aL86wP7F9g1L7bDM7ycxqzKxmwYIFBS9uDRMmwPDh8NJLxf0cERERkUgpBMVsEvcHeql9trvf5O7V7l7dqVOn4lay776wzjrw8MPZjxUREREpgFIIioleu3YZ9q+fclylfHZ+2rWDvfdWUBQREZFGUwpB8b1onek+wK2jdab7CMv1s/M3dCi8+y7MmJH9WBEREZEGKoWg+HS0HmJma9RjZm0JYxkuA4pxc95T0Xq/1B3RsD09gQ+BWUX47PwNjYaGfOSReOsQERGRJiH2oOjuM4GJQDfCzCrJxgFtgL+6+9LERjPrZWa9CvDxzwLvAAPMbFhS+1WEwbsBbnD3OO6PXFu3brDjjrr8LCIiIo2iJKbwA35GmMLvWjMbTAhvfYCBhMu+v0k5/p1ovcZA2Ga2J3BC9HK9aL21md2eOMbdj0n6+VszO5bQs3ivmd0LfAQMBqoJYzhe3cDvVljDhsEll8DChdChQ9zViIiISAWLvUcRvutVrAZuJwTEMUAP4FqgX47zPANsBYyMlkOjbZ2Tto1M89kvA7sT5pgeQpiyrx1wAbBPyc3zPHRoGEvx3/+OuxIRERGpcFYqV1XLXXV1tdfU1BT/g2prYbPNYMAAuOuu4n+eiIiIVBwzm+ru1dmOK4keRclDVVXoVfz3v2HFirirERERkQqmoFiOhg2DJUvg2WfjrkREREQqmIJiORo8GFq10tPPIiIiUlQKiuWoVSvYZ58wnqLuMRUREZEiUVAsV8OGwYcfwptvxl2JiIiIVCgFxXJ1wAFgpsvPIiIiUjQKiuVqo42gd28FRRERESkaBcVyNmwYvPIKzJ0bdyUiIiJSgRQUy9mwaHrqf/0r3jpERESkIikolrPtt4ctt9TlZxERESkKBcVyZhZ6FZ94ApYujbsaERERqTAKiuVu6FBYvjyERREREZECUlAsdwMGQLt2uvwsIiIiBaegWO7WWQf23x8efRRqa+OuRkRERCqIgmIlGDYMPvsMpkyJuxIRERGpIAqKlWC//aBZM11+FhERkYJSUKwE7duHexUVFEVERKSAFBQrxbBh8NZbMGtW3JWIiIhIhVBQrBRDh4b1I4/EW4eIiIhUDAXFStGjB2y3HTz4YNyViIiISIVQUKwkhxwCzz0H8+fHXYmIiIhUAAXFSjJ8eBhL8f77465EREREKoCCYiXZYQfo1QvuvjvuSkRERKQCKChWErPQq/jss/Dpp3FXIyIiImVOQbHSDB8O7nDffXFXIiIiImVOQbHSbL99ePpZl59FRESkgRQUK9Hw4fD88zB3btyViIiISBlTUKxEhx+uy88iIiLSYAqKlWi77cIT0Lr8LCIiIg2goFiphg+HF16AOXPirkRERETKlIJipTr88LC+99546xAREZGypaBYqXr1gh13hHvuibsSERERKVMKipVs+HCYNAk++STuSkRERKQMKShWMl1+FhERkQZQUKxkPXvCzjvr6WcRERGpFwXFSjd8OEyeDB99FHclIiIiUmYUFCudLj+LiIhIPSkoVrqttoJdd9XlZxEREcmbgmJTMHw4vPwyzJ4ddyUiIiJSRhQUmwJdfhYREZF6UFBsCrp3h+pqXX4WERGRvCgoNhXDh8Mrr8AHH8RdiYiIiJQJBcWmInH5WVP6iYiISI4UFJuKbt2gd29dfhYREZGclUxQNLMuZnarmc01s2/MbLaZjTez9nm2s2H0vtlRO3OjdrvU8Z4fmdlEM/vEzJaZ2Swzu8fM+jX8m5WQ4cNh6lSYOTPuSkRERKQMlERQNLMewFTgWGAKcDUwCzgVmGxmHXJspwMwOXrfzKidKVG7U82se5r3XAY8CuwKPA5cA7wKHAhMMrOjG/TlSkni8vM//xlvHSIiIlIWzN3jrgEzmwAMAUa7+3VJ268CTgdudPeTc2jnRuAk4Gp3PyNp+2hCAJzg7vslbd8YmAMsAHZ098+S9g0EngI+cPe1Amaq6upqr6mpyfpdYzdwIMyZA++9B2ZxVyMiIiIxMLOp7l6d7bjYexSjXr4hwGzgjym7xwJLgRFm1iZLO22AEdHxY1N2/yFqf9+UXsUtCOfg5eSQCODuTwNLgE55fJ3SN3IkzJgR5n8WERERqUPsQREYFK0nuntt8g53XwJMAloDfbO00w9oBUyK3pfcTi0wMXo5MGnXDGAF0NvMOia/x8wGAG2BJ3L/KmXg0EOhdWv4y1/irkRERERKXCkExW2i9fQM+2dE656FbsfdFwFnARsBb5vZTWZ2iZndTQiW/wF+muVzy0vbtiEs3nUXLFsWdzUiIiJSwkohKLaL1osz7E9s36AY7bj7eOAQoDlwInA2cDjwMXB76iXpijByJCxeDA8/HHclIiIiUsJKIShmk3jioqFP3aRtx8x+BdwL3A70ANoAuxGeuv67mf0+Y4NmJ5lZjZnVLFiwoIHlNaKBA6FrV11+FhERkTqVQlBM9PS1y7B//ZTjCtaOme0NXAY87O5nuPssd//a3V8FDiY8ET0m3bA6AO5+k7tXu3t1p05l9MxLVRWMGAETJsC8eXFXIyIiIiWqFILie9E60z2IW0frTPceNqSdA6L106kHu/vXhDEYq4Bdsnx2+fnJT6C2Fv7+97grERERkRJVCkExEdKGmNka9ZhZW6A/sAx4KUs7L0XH9Y/el9xOFWEInuTPA2gZrTN1Bya2r8jy2eVnm22gb99w+bkExtIUERGR0hN7UHT3mYQnjLsBP0/ZPY5wz+Bf3X1pYqOZ9TKzXintfAX8LTr+/JR2Tonan+Dus5K2Px+tTzKzzZLfYGb7E0LqcuDFfL9XWRg5Ev77X3jttbgrERERkRJUKjOz9CCEsc7AQ8A7QB/CmIfTgT3cfWHS8Q7g7pbSToeonZ6EWVWmANsSpuP7LGpnZtLxVcAE4AeEwbUfAD6N3nMA4QGY09z9mmzfoWxmZkn2xRewySbw05/CNVm/ooiIiFSIspmZBb7rVawmPHncBxhDeAL5WqBfckjM0s5CwsDb1wJbRe30AW4DdksOidHxtcAPCdMEvk14gGUMYXDvx4B9cwmJZat9exg2DP7xD1hReVfXRUREpGFKokexEpRljyLAv/4FBxwADz4IBx4YdzUiIiLSCMqqR1FitO++sNFGGlNRRERE1qKg2NQ1bw5HHQWPPgoLc7rCLyIiIk2EgqKEp59XroR//jPuSkRERKSEKCgK7Lgj7LyzLj+LiIjIGhQUJRg5Empq4O23465ERERESoSCogRHHhnuV1SvooiIiEQUFCXo3Bn23x/uuAO+/TbuakRERKQEKCjKaiNHwty58MQTcVciIiIiJUBBUVY74IAwW4suP4uIiAgKipKsZctwr+IDD8CiRXFXIyIiIjFTUJQ1nXgiLF+uXkURERFRUJQUO+0E/frBDTeA5gEXERFp0vIOima2o5ldamYPmdkTSdu7mdlwM2tf2BKl0Y0aBdOnw1NPxV2JiIiIxCivoGhmFwCvAr8ChgIDU9r6J3B0waqTeBx+OHToAH/6U9yViIiISIxyDopm9mPgXOA/wM7AJcn73X0WUAMMK2SBEoN114Vjj4UHHwzD5YiIiEiTlE+P4mjgfeBAd38DWJHmmHeArQtRmMTspz8NA2/ffHPclYiIiEhM8gmK3wMmuHu6gJgwF9ioYSVJSdhqKxgyBG66CVatirsaERERiUE+QdGA2izHbAQsr385UlJGjYI5c+DRR+OuRERERGKQT1CcAeyRaaeZNQP2BN5qaFFSIg44ALp00UMtIiIiTVQ+QfFuYFczG5Nh/6+BrYB/NLgqKQ3Nm4cBuCdOhPffj7saERERaWT5BMXxwDTg92b2MrA/gJldEb0eB7wE3FTwKiU+J5wAzZrBjTfGXYmIiIg0spyDorsvI4yb+DdgV6A34b7FM4DdgDuA/dxdTz5Ukk03hYMOgttuC1P7iYiISJOR14Db7r7Y3Y8hPLSyP2Fw7aHAJu4+0t2XFL5Eid2oUbBwIdxzT9yViIiISCMy13y+BVFdXe01NTVxl1Ec7tCrV5it5cUX465GREREGsjMprp7dbbj8pmZ5Vsz+22WY35jZrr0XGnM4OSTYfJkmDYt7mpERESkkeQ7jqLleJxUmpEjw9R+GipHRESkycjrHsUctEcDblemDTeEH/8Y7rgDvvwy7mpERESkETSva6eZDUjZ1C3NNoBmwObAUcB7BapNSs2oUXD77SEs/uxncVcjIiIiRVbnwyxmVgvk+rRLYoq/n7h7kxt0u6IfZklwh+pqWLEC3ngj3LsoIiIiZSfXh1nq7FEELiAERQPOA54Bnk1z3LfAQuBpd383v1KlbJiFXsUTT4RJk2DPPeOuSERERIoo5+FxzOwD4Gp3v7a4JZWnJtGjCLB0KWy2Gey7L9x1V9zViIiISD0UfHgcd99SIVFo0wZOOgnuvRdmz467GhERESmiQj/1LE3B6NFQVQXjx8ddiYiIiBRRtnsU12BmBhwG7AtsBrRMc5i7++AC1CalqksXOPJIuPlmGDsW2rePuyIREREpgpyDopm1BB4D9iY83JJ4yCXBk7ZLpRszBv76V7jhBvj1r+OuRkRERIogn0vPZwEDgd8BnQih8HxgU+BI4GPgTqBFYUuUkrTjjjBkCFx7LXzzTdzViIiISBHkExQPB15197HuvjCx0d0/dfc7gUHAAcBpBa5RStWZZ8Knn8I/mtywmSIiIk1CPkGxBzAp6bUD63z3wn0W8C/gmIJUJqXvBz8IPYtXXBEG4xYREZGKkk9QXMma8zgvIVyCTvYh0L2hRUmZMAu9im+/DY8/Hnc1IiIiUmD5BMVPCE86J0wH+qUcswuwqKFFSRk54ogwAPcVV8RdiYiIiBRYPkFxErBH0usHge+Z2S1m9iMzuxz4AWGaP2kqWrSAU0+Fp56CV1+NuxoREREpoHyC4j+A98ysW/R6PPAKcCzwMDAGmAmcXcD6pBycdBK0bQtXXhl3JSIiIlJA+Uzh94y77+/us6PXXwP9CU9DnwP8H7Czu88pRqFSwtq1gxNPDHM/f/RR3NWIiIhIgTRoCj93X+Xu97n7Ze5+VxQepSk69dSwvuaaeOsQERGRginoXM9m1tzMfl7INqVMbL55eLDlppvgf/+LuxoREREpgIIERQtGEp6EvraebXQxs1vNbK6ZfWNms81svJnlNZGwmW0YvW921M7cqN0uWd73fTO7z8zmRe+bZ2YTzeyH9fk+TdKYMfDVV/DnP8ddiYiIiBRA1qBoZu3N7Ldm9nAUpE4zs3WT9h8AvAXcCmwBPJBvEWbWA5hKeDBmCnA1MAs4FZhsZh1ybKcDMDl638yonSlRu1PNLO0Yj2Z2LvAcMAB4HLgSeARoT5jbWnKx664waFC4/LxiRdzViIiISAM1r2unmXUkBK0tCHM7AxwEDDOzfYAbgOOifY8C57n76/Wo43qgMzDa3a9L+vyrgNOBi4CTc2jnYqAncLW7n5HUzmjgmuhz9kv5jocDFwJPAIe4+5KU/esguTvzTPjhD8ODLSNGxF2NiIiINIB5HVOvmdkVwBnANODvhEA4Atge+A8wBHgZONXdp9SrgNDLNxOYDfRw99qkfW2BedHndnb3pXW00wZYANQCmyQHPjOrij6jW/QZs5K2vw9sBHRz9wX1+Q4A1dXVXlNTU9+3Vw532GEHaNYMpk0Ls7eIiIhISTGzqe5ene24bJee9ydMy9fH3a9w98uBPoRZWvYB7gT2qG9IjAyK1hOTQyJAFPYmAa2Bvlna6Qe0Aial9gpG7U6MXg5M2rUHsCXwGPBFNHD4WWZ2qpmlzjojuUhM6/fmm/DYY3FXIyIiIg2QLSh2Ax5z9+9uOHP3ZYTLzAC/9bq6JHOzTbSenmH/jGjdswjt7B6t5wOvEr7XpYTBxF80s2fNLHU+a8nmqKOgWzcYNy70MIqIiEhZyhYUWxFCVKrPovWsAtTQLlovzrA/sX2DIrTTOVqfTPiuPwDaAjsAEwgPt9yT6QPN7CQzqzGzmgUL6n3VuvK0aAG/+Q288op6FUVERMpYQwfcbozuosRNbg39rHTtNEvad5i7P+nuX7n7W8DBhEvse2W6DO3uN7l7tbtXd+qkjsc1jBwZehXPP1+9iiIiImWqzqeeIzub2U9StwGY2QhWB7DvuPtf86gh0dPXLsP+9VOOK2Q7X0TrWe4+Lflgd19mZhOA44HehGF3JFfrrAPnngsnnBB6FX/0o7grEhERkTzlEhQPjJZUBtye4T35BMX3onWmexC3jtaZ7j1sSDuJ92SaSiQRJFtl+WxJ5yc/gYsuCr2KP/yhnoAWEREpM9mC4l8aoYano/UQM6tKMzxOf2AZ8FKWdl6KjutvZm3TDI8zJOXzIAyyvQrY2sxaJD+0E9khWs/O4/tIQqJX8fjj1asoIiJShuocR7HRigiXeIeQecDtG9395KTtvQDc/d2Udm4ETgKucvcxSdsTA25PcPfUAbfvAI4CLnL3c5O270N4oOVLwhiLdU5grHEUM1i5ErbZBjp0gClT1KsoIiJSAnIdR7FUgmIP4EXCU8gPAe8QxmscSLhUvIe7L0w63gHc3VLa6RC10xN4ijCrzLaES+efRe3MTHlPZ8JYjVsBz7N6JpqDCQ++HOnuGZ98TlBQrMOtt4ZexUcegQMOiLsaERGRJq+sgiKAmXUFLiBMsdeBMCPLg8A4d1+UcmzaoBjt2xAYS5hqcBNgIfBvwvSCn2T47A2BcwnhcDNgCfACcIm7Z7vkDSgo1mnlSujVC9q3D0PmqFdRREQkVmUXFMudgmIWt90Gxx2nXkUREZESUKgp/EQK4+ijoXt3jasoIiJSRhQUpXEknoCeOhUefTT78SIiIhI7BUVpPCNGQI8e6lUUEREpEwqK0niaNw+9iq++ql5FERGRMqCgKI3r6KPVqygiIlImcpnCDwAz2zyHw2qBL939y/qXJBUt0at47LGhV3Ho0LgrEhERkQxyHh7HzGoJA1DnYj5wH2EMxM/rWVtZ0fA4eVi1CrbdFtq0CQ+3NGsWd0UiIiJNSjGGx/krYW5kAxYDzwJ3R+vF0fZngceAFcDPgVfMrFN+pUvFa94cLrwQpk2DO+6IuxoRERHJIJ+geAmwE3Ap0NXdB7n7/7n7IKAr8Pto/xigOzCOMBXerwtbslSEI46A3r3DZeivv467GhEREUkjn6B4KTDN3c9x96XJO9x9qbufDbwBXOrute4+Dngd0E1osjYzuOIK+OQTGD8+7mpEREQkjXyC4gDgxSzHvAjslfT6JaBLvkVJE/H978PBB8Mll8D8+XFXIyIiIinyCYotgY2zHLNJdFzCV8CqfIuSJuTSS2H5chg3Lu5KREREJEU+QXEacISZ7ZBup5ntCAwnXG5O6AYsqHd1Uvl69oSTT4abboJ33427GhEREUmST1C8AFiX8CTzn83sGDPbP1rfDLwc7b8QwMxaAUOASYUuWirMeeeFoXLOOivuSkRERCRJzgNuu/sEMzsKuAE4HjguaXdiyJzj3H1CtK0FcATwXoFqlUrVqROccw6cfTY88wzsvXfcFYmIiAh5DLj93RvM1gMOBHYB2gFfAq8BD7n7koJXWCY04HYDLVsG22wDnTvDlClQpdklRUREiiXXAbdz7lFMcPevgL9Hi0hhtGoFF18MI0bAP/8JRx0Vd0UiIiJNnrptpHQceSTsumu4DL1sWdzViIiINHl59yiaWTXQG2gPpJuk1939woYWJk1QVVUYhHvQILj2Wj3cIiIiErOc71E0s/WB+4GBhIdXMnF3TxcgK5ruUSygoUPhuedg5kzo2DHuakRERCpOMe5RvBwYBDwP3AZ8jAbTlmL4/e/he9+DCy4IPYsiIiISi3yC4oHAq8BAd68tUj0isO22cOKJ8Kc/wSmnhEG5RUREpNHl8zBLO+BphURpFOefD61bwy9+AXkO4SQiIiKFkU9QnAFsVKxCRNaw0Ubwu9/BxIlw771xVyMiItIk5RMU/wgMNbPNilWMyBpGjYJddoHTToMvv4y7GhERkSYnn6D4b2AiMMnMjjWzHc1s83RLkWqVpqZ5c7jhBpg3D8aOjbsaERGRJiefh1lmA04YGufmOo7zPNsVyax3b/jpT8PTzyNHws47x12RiIhIk5FPoPsrIQSKNK6LL4b77oOf/QxeeEHzQIuIiDSSnIOiux9TxDpEMmvfPszYMnIk3HJLGDpHREREik5dM1IeRoyAAQPCtH4LFsRdjYiISJOgoCjlwQyuvx6WLNEc0CIiIo0k46VnM7uVcE/iOe4+P3qdC3f34wtSnUiy7beHMWPgssvguONgzz3jrkhERKSimWeY9cLMaglBcVt3nx69zoW7e7NCFVguqqurvaamJu4yKt/SpbDddrD++vDqq7DOOnFXJCIiUnbMbKq7V2c7rq6HWbaM1nNSXovEp02bMFTOQQfBNdfAmWfGXZGIiEjFytijKPlRj2IjGzYMnnoK3nkHunaNuxoREZGykmuPoh5mkfJ07bVQWwunnhp3JSIiIhUr7xlUzKwZsA3QHkh7L6K7P9fAukTq1q0bnHce/PrXcPfdMHx43BWJiIhUnLwuPZvZb4HTgXZ1HaeHWaRRrFoFe+wBs2bBW2/BRhvFXZGIiEhZKMTDLKkN/goYBywG/gZ8DKyqd4UiDdW8OfzlL7DLLmE+6AceCOMtioiISEHkc+n5RMIT0Lu6u6bGkNKw7bZw0UXh6ec77ggzuIiIiEhB5PMwS1fgQYVEKTmnnQb9+8MvfgFz5mQ/XkRERHKST1CcTz0efhEpumbN4PbbYeVKOOEE0JBPIiIiBZFPULwb2MfMWharGJF622qrMLXf44/DLbfEXY2IiEhFyCcongfMA+41M83SIqXnZz+DgQPhjDPgww/jrkZERKTs5RMU3wK6AT8E3jezRWY2K80ysz6FmFkXM7vVzOaa2TdmNtvMxptZ+zzb2TB63+yonblRu11yfP8IM/NoOaE+30ViUlUFt94aLj0fd1wYkFtERETqLZ+gWEUYDuejaFkMWJol79lezKwHMBU4FpgCXA3MAk4FJptZhxzb6QBMjt43M2pnStTuVDPrnuX9XYHrgK/y/Q5SIrp1g6uuCtP7/elPcVcjIiJS1nJ+OMXduxWxjuuBzsBod78usdHMriIM8H0RcHIO7VwM9ASudvczktoZDVwTfc5+6d5oZgbcBiwE7gfOrNc3kfidcALcfz/86lew777h/kURERHJW869f2a2uZltXOgCol6+IcBs4I8pu8cCS4ERZtYmSzttgBHR8WNTdv8han/fOnoVRwODCL2PS3P/BlJyzODmm6FFCzjmGPj227grEhERKUv5XCb+gNBjV2iDovVEd1/jpjJ3XwJMAloDfbO00w9oBUyK3pfcTi0wMXo5MPWNZrYtcClwjeaprhCbbQbXXguTJoVL0SIiIpK3fILi/4DPi1DDNtF6eob9M6J1z2K0Y2bNCVMSfgSck+UzpJwcfTQceiiccw689FLc1YiIiJSdfILiS8AuRaihXbRenGF/YvsGRWrnPML3Osbdl2X5DCkniUvQXbvCEUfAokVxVyQiIlJW8gmK5wPfj2HIGIvWDZ1uY612zKw3oRfxSnefnHeDZieZWY2Z1SxYoJkNS9IGG8Bdd8G8eXDssZq1RUREJA/5TMm3P/AMcKOZjSIMO/Mpawc4d/cL82g30dPXLsP+9VOOK0g7SZecpwO/zV7m2tz9JuAmgOrqaiWQUrX77nD55WFO6PHj4fTT465IRESkLOQTFM9P+nkXMl+GdiCfoPhetM50D+LW0TrTvYf1bWe9pGOXh9Fx1vJnM/sz4SGX07J8vpSy0aPhmWfgrLOgf3/o3TvuikREREpePkFxraeFC+TpaD3EzKqSn3w2s7ZAf2AZ4R7JurwUHdffzNomP/lsZlWEIXiSP+8bINOkwLsSgvALhACa92VpKTFmYdaWXXYJ9yu++iq0z2vSHxERkSYnnwG3ny1GAe4+08wmEoLczwkzoySMA9oAN7r7d2Mbmlmv6L3vJrXzlZn9DTiJ0Ps5JqmdUwjTD05w91nR8cuAtPdbmtn5hKD4F3e/uWHfUEo8y0NZAAAgAElEQVRG+/bhfsU99wxT/N1/fwiQIiIiklY+PYrF9DPgReBaMxsMvAP0IfRiTgd+k3L8O9E69a/8OcDewBlmtjPhPsptgQOBzwhBVJqyPn3gsstgzBi47rpwSVpERETSynte5mJw95lANXA7ISCOAXoA1wL93H1hju0sJAy8fS2wVdROH8LUfLtFnyNN3emnw9ChcOaZ8MorcVcjIiJSsszzGC7EzDYBzgX2BTYDWqQ5zN29VHoqG011dbXX1NTEXYbkatGicL9is2bhfsUNsg3TKSIiUjnMbKq7V2c7Lp+5njcDaoCfEuZCbkmYzWQG8C3hMvA04Pn6FCzSqDbcEO68Ez7+GI4/XuMrioiIpJHPpefzgI2B/dx9p2jbbe7eC+gOTCDMtXxIYUsUKZJ+/eCSS8JDLb//fdzViIiIlJx8guK+wOPu/kTqDnf/BDicEBTHFag2keIbMyYMl/PrX8Mjj8RdjYiISEnJJyhuDLyV9PpbQjAEwvA0wH8ITxiLlIfE+Iq77gpHHgn//W/cFYmIiJSMfILil6z58MoXhAdaki0GOjW0KJFG1bo1PPQQrLceDBsGn38ed0UiIiIlIZ+g+CHQNen1NGCQmbWGNWY/+aRw5Yk0ks02gwcfhLlz4bDDYMWKuCsSERGJXT5B8UlgoJmtE73+C7Ap8KKZXQ5MArYH7ipsiSKNpE8fuOUWePZZ+MUv9CS0iIg0efmMd3gL4XJzR2Ceu99hZrsBvwB2jI65E7iosCWKNKKjjgr3KV56KXzve3DKKXFXJCIiEpu8BtxO24BZJ8LwOLPdfX5BqipDGnC7gtTWwkEHwWOPwYQJMHhw3BWJiIgUVMEH3M7E3Re4+8tNOSRKhamqgr//HbbdFg4/HGbMiLsiERGRWNQrKJpZLzM72MxGFLogkZLQti08/HAIjcOGweLFcVckIiLS6PIKima2s5nVEMZTvBe4PWnfXmb2tZkNLWyJIjHZcku47z54/30YPlxPQouISJOTz1zPPYFngG2Aa4B/pxzyHLAIOKxQxYnEbq+94MYbYeJEOPbYcP+iiIhIE5HPU89jCQNu7+bu75jZWGD/xE53dzObDOxe4BpF4nXccfDZZ2Gav44dYfz4MKOLiIhIhcsnKA4G7nf3d+o45iNgn4aVJFKCzjoL5s8PIXGjjeCcc+KuSEREpOjyCYobkH3WlSrWnOZPpDKYwZVXwoIF8JvfQOfOcMIJcVclIiJSVPkExc+ArbIcsz3wcf3LESlhVVVw222waBH89KfQoQMcfHDcVYmIiBRNPk89PwUMNbNt0u00s90Jl6cnFKIwkZK0zjpwzz3Quzf83/+F6f5EREQqVD5B8RJgFfCcmY0izPOMmW0fvX4EWAJcUfAqRUpJmzbw6KPQvXsYY/H11+OuSEREpChyDoru/h5wKOEexD8AJwAGvAH8Mdp+iLt/VIQ6RUpLhw5her927WC//WDmzLgrEhERKbi8Btx298eBLYEzgLuBJ4D7gV8CW7n7UwWvUKRUde0awuLKlTBkCMybF3dFIiIiBWXuXrjGzNYFWrj7lwVrtExUV1d7TU1N3GVIHF5+GQYPhi5d4KmnYNNN465IRESkTmY21d2rsx1Xr7me6/AnwuwsIk1Hnz7w+OMwZw7svXdYi4iIVIBCB0UI9y2KNC177hnC4qefhrD4SbYhR0VEREpfMYKiSNPUv3+4Z3H+/BAWP9aQoiIiUt4UFEUKqV8/mDgxzOCy997wkQYBEBGR8qWgKFJoffvCf/4DCxeGsPjhh3FXJCIiUi8KiiLF0Ls3PPEEfPFFCIuzZ8ddkYiISN4UFEWKpbo6hMXFi2GvveCDD+KuSEREJC91BkUz+zafBfhJI9UtUh522y2ExSVLYMAAePvtuCsSERHJWbYeRavHIiLJdt0Vnn4aVq0Kw+hMmhR3RSIiIjmpMyi6e1U9lmaNVbxI2dhpJ3jxRejYEX7wA3jwwbgrEhERyUr3KIo0li23DGFxp53g0EPhhhvirkhERKROCooijaljR3jySdhvPxg1Cs47Dwo437qIiEghKSiKNLY2beChh+C44+DCC+HEE8P9iyIiIiWmedwFiDRJzZvDzTfDppvC734Xpv276y5o3TruykRERL6jHkWRuJiFHsXrr4d//QsGDYLPP4+7KhERke8oKIrEbdQouPdeeP31MKPLm2/GXZGIiAigoChSGg45BJ59FpYvh3794P77465IREREQVGkZPTpAzU1sMMOYficsWOhtjbuqkREpAlTUBQpJZtuCs88A8ccAxdcEHoalyyJuyoREWmiFBRFSs2668Ktt8I118Cjj4ZL0e+/H3dVIiLSBCkoipQiMxg9GiZMgHnzYPfdYeLEuKsSEZEmRkFRpJQNHgyvvAJdu8L++8NVV2kmFxERaTQKiiKlrnv3MEf0wQfDmDHhQZdFi+KuSkREmoCSCYpm1sXMbjWzuWb2jZnNNrPxZtY+z3Y2jN43O2pnbtRulzTHdjCzE8zsATN738yWmdliM3vBzI43s5I5P9LErbce3HMPXHlluG9x551h0qS4qxIRkQpXEkHIzHoAU4FjgSnA1cAs4FRgspl1yLGdDsDk6H0zo3amRO1ONbPuKW85HPgz0Ad4GRgP3AfsANwM3G1m1qAvJ1IoZnDGGSEgrrMO7LUXXHQRfPtt3JWJiEiFKomgCFwPdAZGu/tB7n62uw8iBL1tgItybOdioCdwtbsPjto5iBAcO0efk2w6MAzo4u5Hufuv3f04oBfwMXAocEhDv5xIQe2+O7z2Ghx+OJx7LgwZEh54ERERKbDYg2LUyzcEmA38MWX3WGApMMLM2mRppw0wIjp+bMruP0Tt75vcq+juT7n7I+6+xqjG7v4pcEP0cu88vo5I41h/ffjHP+CWW2DyZNhpJ3j88birEhGRChN7UAQGReuJaQLbEmAS0Brom6WdfkArYFL0vuR2aoHE2CIDc6xrZbRelePxIo3LDI47DqZOhY03Dk9F//KXsGJF3JWJiEiFKIWguE20np5h/4xo3bOR2sHMmgM/iV6qm0ZK27bbwssvw6hRcMUVYYDuN9+MuyoREakApRAU20XrxRn2J7Zv0EjtAFxKeKDlMXefkOkgMzvJzGrMrGbBggU5NCtSJK1awfXXw/33w8cfw267we9+BytXZn+viIhIBqUQFLNJPHXc0FGGc2rHzEYDY4B3Cfc8ZuTuN7l7tbtXd+rUqYHliRTAwQfDW2+FsRZ/+1vo0wfeeCPuqkREpEyVQlBM9PS1y7B//ZTjitaOmf0cuAZ4Gxjo7hrVWMpPp07wz3/CfffBnDlQXQ0XXKDeRRERyVspBMX3onWmewe3jtaZ7j0sSDtmdhrh6ej/EkLip1k+T6S0HXJI6F087DAYOxZ694Zp0+KuSkREykgpBMWno/WQ1JlQzKwt0B9YBryUpZ2XouP6R+9LbqeKMARP8ucl7z+LMGbj64SQ+Fm+X0KkJHXsGIbRuf/+MNZidTWMG6cno0VEJCexB0V3n0kYuqYb8POU3eOANsBf3X1pYqOZ9TKzXintfAX8LTr+/JR2Tonan+Dus5J3mNlvCQ+vTAUGu/vnDftGIiUoce/iEUfA+efDjjvCE0/EXZWIiJQ4c2/oMyIFKCJM4fciYfaUh4B3CNPqDSRcKt7D3RcmHe8A7m4p7XSI2ukJPEWYvm9b4EDgs6idmUnHjwRuB74FriP9/Yuz3f32bN+hurraa2pqcvq+IrF67DEYPRpmzoThw8P80V3WmgpdREQqmJlNdffqrMeVQlAEMLOuwAXAfkAHYB7wIDAu9aGSTEEx2rchYWaWg4BNgIXAv4Hz3P2TlGPPZ+1ZXFI96+57Z6tfQVHKyvLlcPnlcPHF0KxZuIfx1FOhRYu4KxMRkUZQdkGx3CkoSln64AM47TR4+OEwcPcf/gCDBmV/n4iIlLVcg2Ls9yiKSIy23BIeeggeeQS++QYGD4Yf/zgMqyMiIk2egqKIwAEHhIddxo0LwbFnTzjvPFiyJPt7RUSkYikoikiw7rohHL71FgwdChdeCD16wB//qMG6RUSaKAVFEVlT9+5w550wZQpsvz2ccgpstx3ccw/onmYRkSZFQVFE0tt9d3jqKfjXv0Jv4/Dh0LcvPPts3JWJiEgjUVAUkczM4Ic/hNdfh1tvDQ+57L13uDT9xhtxVyciIkWmoCgi2TVrBsceCzNmwKWXwvPPw047hfmkX3st7upERKRIFBRFJHetWsFZZ4XxF8eODZemd90Vhg2DV16JuzoRESkwBUURyV/79mHO6A8/DE9Hv/AC9O4dLlNPnhx3dSIiUiAKiiJSf+3awbnnhsB4ySWhV3GPPWDIkHB5Wk9Ji4iUNQVFEWm4tm3h7LPDJenLL4dp02DAAOjXD+6+G1atirtCERGpBwVFESmc9daDM88MgfG662DhQjjiCNhqK7jqKli8OO4KRUQkDwqKIlJ4rVuHgbrffRcefBC22ALGjIGuXeGMM2D27LgrFBGRHCgoikjxNGsGBx4YBul+5ZUw/uK114apAYcPh0mTdB+jiEgJU1AUkcZRXQ1//3u4LH3mmTBxIuy5J+y4I/zhD/C//8VdoYiIpFBQFJHG1bUrXHYZfPIJ3HQTtGwJv/gFbLopHHccvPyyehlFREqEgqKIxGO99eDEE6GmJixHHx2ekO7bNwzifcMNsGRJ3FWKiDRpCooiEr/ddgu9i3PnwvXXhx7FUaNgk03gmGPCDDC1tXFXKSLS5CgoikjpWH/9EBBfew1eegl+/GN44AEYPDg8OX322fDWW3FXKSLSZCgoikjpMYM+feDmm+HTT+HOO2GnneCKK2CHHUIP5PjxMH9+3JWKiFQ0BUURKW2tWoVBux99FObMCQHRDE4/HTbbDPbfH269FRYtirtSEZGKo6AoIuVjo43g1FPDwy9vvQW//GUY1Pv448O+/fYLvZCffx53pSIiFUFBUUTK03bbwSWXwKxZYTDvMWNgxozwJPXGG8M++4QHZD77LO5KRUTKlrnGKyuI6upqr6mpibsMkabNHV5/He69F+65JwTHqqowsPcBB4SlV69w6VpEpAkzs6nuXp31OAXFwlBQFCkx7vDmmyE0PvwwTJsWtvfosTo0DhgALVrEW6eISAwUFBuZgqJIifv44/BAzKOPwpNPwjffQNu2sO++ITQOGRLGbRQRaQIUFBuZgqJIGVm6NITFRHCcNy9s32GHcG/jPvuE3sY2beKtU0SkSBQUG5mCokiZqq0Nl6X/85+wPP986G1s0QL22CP0NO6zD+yyCzRrFne1IiIFoaDYyBQURSrEsmUhLCaCY+Lexg02gO9/P/Q07rVXCI7Nm8dbq4hIPeUaFPWvnIhIslatQi/ikCHh9fz54TL100/Dc8/BI4+E7eutB/37h9A4YADsvrsejBGRiqMexQJRj6JIEzFvXgiMzz0Hzz67eu7pddcNYbFvX+jXLywbbxxvrSIiGejScyNTUBRpoj7/PFyqfu45eOklePVVWLEi7Ntii9WhsW9f2Hln9TqKSElQUGxkCooiAsDy5fDaayE0Tp4c1h9/HPa1aAE77gi77bZ62WEHhUcRaXQKio1MQVFEMpozJwTGl1+GqVPDsnhx2JcaHnfeGbbfHlq3jrdmEaloCoqNTEFRRHLmHuaorqlZHRyTw2NVFWy9dQiQO+20et21q6YfFJGCUFBsZAqKItIgtbUhPE6bBm+8EZZp0+CDD1Yf065dCI3bbbfmsskmCpAikhcNjyMiUk6qqmCrrcJy6KGrt3/5Jfz3v2sGyLvugv/9b/Ux7dqtGRx79YKePaFbN431KCINoh7FAlGPoog0GvcwvuPbb6+9LFiw+rjmzaFHjxAak5ett4ZNN1UvpEgTph5FEZFKZRbGaNx4Yxg0aM19CxbA9OlrLxMnhqkJE1q1gi23hO7dQ5hMXnfrFvaLSJOnoCgiUkk6dQpL//5rbq+tDcP0JILjrFlhmTkzzDqzdOmax2+ySQiMW2yRfllvvUb7SiISHwVFEZGmoKpqdcjbZ58197mHnshEcJw1KzxE8+GHMGUK3HcfrFy55ns23BA23xy6dFlz6do1rDfbDNq0abzvJyJFoaAoItLUmUHnzmHp23ft/bW1YerCDz9cc/n447BMngwLF679vvbtQ8/kJpuEeyKT14mfN95YgVKkhCkoiohI3aqqQg/hZpvBHnukP2bZsjCw+CefrLnMm7d6fux581ZPb5isTRvYaKPMS+JyeqdOoSezqqq431dEvqOgKCIiDdeq1erhfTJxh0WLVofHuXPh00/DE9yJ5f33YdKkMId2ulE5qqqgQ4cQGjt2XL3u0CH9suGGoWezWbPifXeRClYyQdHMugAXAPsBHYB5wIPAOHf/Io92NgTOAw4CNgEWAo8D57n7J8X8bBERqYPZ6gC3ww51H7tqVQiL8+eH+yczLYkhgRYtCpfIM31uu3YhMNa1tGsHG2yw9rpVKw0lJE1WSQRFM+sBvAh0Bh4C3gV6A6cC+5lZf3dPcwPMWu10iNrpCTwF3An0Ao4FfmRm/dx9VjE+W0RECqh589VDAOWitjYMTr5w4epl0aI1f/7ii9XLnDmrf053OTy1lkRwXH/9sLRtu/rn5KVt2/BEePI6+eeWLRU6payURFAEricEtdHufl1io5ldBZwOXAScnEM7FxNC4tXufkZSO6OBa6LP2a9Iny0iInGpqgphboMNwniQuXIP91d+8UWYa/t//wvrdD9/+eXqZc4ceOed1a+Tx6isS7Nm4Z7M9dZbvU7+uU2bsLRuvXqd+nNiadUqLMk/t2qleziloGKfmcXMugMzgdlAD3evTdrXlnAZ2IDO7r40bSPh2DbAAqAW2MTdlyTtq4o+o1v0GbMK+dmgmVlERJq0FStCYPzqK1iyJCyZfl66dPXy1Vdrr7/+evVSHy1bhsC47rrZ1y1bhnW6n+taWrRY++cWLVYvLVvqvtASV04zsySmFZiYHNQA3H2JmU0ChgB9gSfraKcf0CpqZ0nyDnevNbOJwEnAQCBx+blQny0iIk1ZixbhoZqOHQvXZm0tLF8eAuPSpavD49KloRd02bLwOnmd/PPy5enXX3wR1t98E7YtXx5+XrYs832e9VFVtWZ4XGedzD+nLum2N29e98/Nm69eUl/XtTRrtvbPda1TlwrvwS2FoLhNtJ6eYf8MQljrSd1hLZd2iNop9GeLiIgUVlXV6svMhQygdVm1as0AuWJFeJ1pWbFizSV5W+LnlSvXPCbd65Urw+clfk7evnJlqCv150KG2oZKFyBTw2SmbcnrxM+nnAJHHx33twJKIyi2i9aLM+xPbN+gCO0U6rNFRETKX6JnrRwGQa+tXTNAfvvt6iCZvCTv//bbtfevWrX6vcnHpK4TP+e71NZm3pa8Tv65Zcu4z+53SiEoZpN4PKyhN1PWp50632NmJxEuZ7P55pvXvzIRERHJT/KlbSmaUriwnui1a5dh//opxxWynQZ9trvf5O7V7l7dqVOnLOWJiIiIlJdSCIrvReueGfZvHa0z3UfYkHYK9dkiIiIiFacUguLT0XpINIzNd6IhavoDy4CXsrTzUnRc/+h9ye1UER5KSf68Qn62iIiISMWJPSi6+0xgImGMw5+n7B4HtAH+mjyOoZn1MrNeKe18BfwtOv78lHZOidqfkDwzS30+W0RERKSpiH3AbUg7jd47QB/CmIfTgT2Sp9EzMwdwd0tpJ3UKvynAtsCBwGdROzMb8tmZaMBtERERKRe5Drgde48ifNezVw3cTghpY4AewLVAv1znWo6O6xe9b6uonT7AbcBuqSGxkJ8tIiIiUmlKokexEqhHUURERMpFWfUoioiIiEjpUVAUERERkbQUFEVEREQkLQVFEREREUlLQVFERERE0lJQFBEREZG0NDxOgZjZAuDDIn9MR+DzIn9GU6TzWhw6r8Wh81ocOq/FofNaHIU4r1u4e6dsBykolhEzq8llzCPJj85rcei8FofOa3HovBaHzmtxNOZ51aVnEREREUlLQVFERERE0lJQLC83xV1AhdJ5LQ6d1+LQeS0Ondfi0HktjkY7r7pHUURERETSUo+iiIiIiKSloCgiIiIiaSkoljgz62Jmt5rZXDP7xsxmm9l4M2sfd22lzswOM7PrzOx5M/vSzNzM7sjynj3M7DEzW2RmX5vZG2Z2mpk1a6y6S5mZdTCzE8zsATN738yWmdliM3vBzI43s7T/pui8Zmdml5nZk2b2cXReF5nZa2Y21sw6ZHiPzms9mNmI6N8DN7MTMhxzgJk9E/1+f2VmL5vZyMautVRFf4s8w/Jphvfo9zVHZvZ9M7vPzOZFf/vnmdlEM/thmmOLel51j2IJM7MewItAZ+Ah4F2gNzAQeA/o7+4L46uwtJnZ68BOwFfAJ0Av4O/ufnSG4w8E7gOWA3cBi4ChwDbAve5+eGPUXcrM7GTgT8A84GngI2Aj4BCgHeH8He5J/7DovObGzFYArwJvA58BbYC+QDUwF+jr7h8nHa/zWg9m1hV4E2gGrAec6O43pxxzCnAdsJBwblcAhwFdgCvd/cxGLboEmdlsYANgfJrdX7n7FSnH6/c1R2Z2LnAhYUDtRwn/3nYEdgGedvdfJR1b/PPq7lpKdAEmAA78ImX7VdH2G+KusZQXQqDeGjBg7+ic3ZHh2PUJf5y/AaqTtq9LCOsO/Dju7xT3AgyK/hGqStm+MSE0OnCozmu9zu26GbZfFJ2n63VeG3yODXgCmAlcHp2nE1KO6Ub4o7sQ6Ja0vT3wfvSefnF/l7gXYDYwO8dj9fua+3k9PDof/wHaptm/TmOfV116LlFm1h0YQviP8Y8pu8cCS4ERZtamkUsrG+7+tLvP8Oi/nCwOAzoBd7p7TVIby4Fzo5ejilBmWXH3p9z9EXevTdn+KXBD9HLvpF06rzmKzkk6d0frrZO26bzWz2jC/+wcS/g3NJ3jgJbAH9x9dmKju38BXBy9PLmINVYi/b7mILp15zLga+BId1+Seoy7r0x62SjntXlDG5CiGRStJ6b5o7zEzCYRgmRf4MnGLq4CJc7342n2PUf4D3cPM2vp7t80XlllJfEP2KqkbTqvDTc0Wr+RtE3nNU9mti1wKXCNuz9nZoMyHFrXuf13yjFNXUszOxrYnBC83wCec/dvU47T72tu9gC2BO4FvjCzHwE7EHq4p7j75JTjG+W8KiiWrm2i9fQM+2cQgmJPFBQLIeP5dvdVZvYBsD3QHXinMQsrB2bWHPhJ9DL5Hy2d1zyZ2ZmEe+faEe5P3JPwB/jSpMN0XvMQ/X7+jXB7xDlZDq/r3M4zs6VAFzNr7e5fF7bSsrMx4bwm+8DMjnX3Z5O26fc1N7tH6/mE+5W/l7zTzJ4DDnP3BdGmRjmvuvRcutpF68UZ9ie2b9AItTQFOt8Ncynh/3wfc/cJSdt1XvN3JuH2ktMIIfFxYEjSHwfQec3XeYQHAY5x92VZjs313LbLsL+puA0YTAiLbQih5kbCPZ7/NrOdko7V72tuOkfrk4FWwA+AtoR/WycAA4B7ko5vlPOqoFi+LFrrsfXGofOdgZmNBsYQnsofke/bo7XOa8TdN3Z3I/wBPoTQG/Came2aRzM6rxEz603oRbwyzaW7ejUZrZv0uXX3cdE9y/Pd/Wt3/6+7n0x42LIVcH4ezemcBonhbIzQc/iku3/l7m8BBxNG79jLzPrl2F5BzquCYunK9n+t66ccJw2j810PZvZz4BrCkC4D3X1RyiE6r/UU/QF+gHCLSQfgr0m7dV5zkHTJeTrw2xzfluu5/bIBpVWyxENtA5K26fc1N19E61nuPi15R9QTnrha0ztaN8p5VVAsXe9F654Z9ieegMx0D6PkJ+P5jv7YbEl4SGNWYxZVyszsNOAPwH8JITHdILs6rw3k7h8Sgvj2ZtYx2qzzmpv1COdoW2B58qDQhMv7AH+OtiXGA6zr3G5CuMz6ie5PzOizaJ08Iod+X3OTOE//y7A/ESRbpRxf1POqoFi6no7WQ1JnuzCztkB/YBnwUmMXVqGeitb7pdk3AGgNvNjEn8j7jpmdBVwNvE4IiZ9lOFTntTA2jdaJp0l1XnPzDXBLhuW16JgXoteJy9J1ndv9U46RtSUuiyaHE/2+5uY5QrDb2sxapNm/Q7SeHa0b57zGPbikljoH3tSA24U7l3uTfcDtBWhA2FzO5W+j81EDbJjlWJ3X3M5pL2DjNNurWD3g9iSd14Ke8/NJP+D2lmjA7Wznbvt0/+0DWxBG5HDgnKTt+n3N/dzeEZ2P36Vs3weoJfQ2btCY51VT+JWwNFP4vQP0Icw4Mh3YwzWFX0ZmdhBwUPRyY2Bfwv/lPh9t+9yTpuKKjr+X8EfiTsJUSMOIpkIChnsT/w8mmuv2dkLP1nWkv/dltrvfnvQendcsosv4lxN6FGYSQspGwF6Eh1n+v727DbWsquM4/v0xMtYLnTGofEAQUrSI0kBGctARJZQYbcAYepoh7IUEgYKChOCVCHwTgi+kFyEi5fjwQvOpIBHHHCITHRJ8KnBEHM3n0R50UP+9WPvA9rDPvXP0Ho/X+/3AZnHWXnudtdc999z/XXuvtV8Ezqyqx3vH2K8fQZIF2uXnoUf4/Qy4Bh/hN6jru8toV76eAd4CvgR8mxak3ANsqar9vWP8vB6AJF8AdgHH0v5WPUQLwLfQAr/vV9WtvfKz79d5R89uS/53cTRtGYIXaF9Wz9ImDyw6kuP2gRGDSduegWNOpX3JvU67tP8YcDGwZt7n80nYDqBPC7jffp26X79KewLTbtrzXd+lBeF/6/p88Pfdfv1IfT76LP9kwv7NwE5aEPSf7mexfd7t/iRstH9gdtBWOniDttj+y7THzm2DNgg1cJyf1wPr38/Rrhw+0/3df5U2WHTKPPrVEUVJkiQNcjKLJOUkm0UAAAOtSURBVEmSBhkoSpIkaZCBoiRJkgYZKEqSJGmQgaIkSZIGGShKkiRpkIGiJEmSBhkoStIqkWQhSSXZNO+2SFoZDBQl6QB1QdZS26Z5t1OSlstB826AJK1AVy6yb8/H1QhJmjUDRUmaUlUtzLsNkvRx8NKzJM1I/57AJNuTPJrkf0leSnJdksMnHHdckhuSPJ9kf5K93evjJpRfk+TCJLuS7Ove459JfrPIMecneSjJf5O8luSmJEct5/lLWvkcUZSk2bsY+BZwM/BHYCPwY2BTkg1V9fKoYJKTgXuBQ4A7gMeBE4AfAOclObOqHu6VXwvcDZwFPAfcCLwJHANsAR4E/jHWnp8C53b17wQ2AFuBryc5sareWc6Tl7RyGShK0pSSLEzY9XZVXTWQfw6woaoe7dVxNXARcBVwQZcX4AbgUOCHVfW7XvmtwE3Ab5N8pare73Yt0ILEO4Hv9oO8JAd3dY07Gzi5qh7rlb0R+B5wHnDLxJOXtKqkqubdBklaEZIs9YW5r6rW98ovAFcA11XVBWN1rQOeBQ4G1lfVO0lOpY0A/qWqvjnw/n+mjUaeXlUPJFkDvAqsBY6tqr1LtH/Unl9W1eVj+84A7gN+VVWXLHGeklYJ71GUpClVVSZs6yccsnOgjn3AbuAzwJe77G906X0T6hnln9SlJwDrgL8vFSSOeXgg77kuPWyKeiR9yhkoStLs/WtC/otdum4sfWFC+VH++rH0+Snb88ZA3rtdumbKuiR9ihkoStLsfXFC/mjW876xdHA2NHDEWLlRwOdsZUkzYaAoSbN3+nhGd4/iicDbwBNd9miyy6YJ9YzyH+nSJ2nB4teSHLkcDZWkPgNFSZq9HyU5aSxvgXapeUdvpvIu4ClgY5Lz+4W716cBT9MmvFBV7wHXAp8Fft3Ncu4fszbJ55f5XCStIi6PI0lTWmR5HIDbq2r3WN4fgF1JbqHdZ7ix2/YAl40KVVUl2Q78Cbg5ye9po4bHA98B3gK29ZbGgfY4wQ3AZuDpJHd15Y6mrd14KXD9hzpRSauegaIkTe+KRfbtoc1m7rsauI22buJW4N+04O3nVfVSv2BV/bVbdPty2vqIm4FXgB3AL6rqqbHy+5OcDVwIbAO2AwH2du/54PSnJ0mN6yhK0oz01i08o6run29rJGl63qMoSZKkQQaKkiRJGmSgKEmSpEHeoyhJkqRBjihKkiRpkIGiJEmSBhkoSpIkaZCBoiRJkgYZKEqSJGmQgaIkSZIG/R8z4DakqQyZYAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(1, 1, figsize=(10,6))\n",
    "ax.plot(exponential_decay_model_history.history['lr'] ,'r') #, label='learn rate')\n",
    "ax.set_xlabel(r'Epoch', fontsize=20)\n",
    "ax.set_ylabel(r'Learning Rate', fontsize=20)\n",
    "#ax.legend()\n",
    "ax.tick_params(labelsize=20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAn0AAAGJCAYAAAD/mIVfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3XmcXFWd9/HPr7PvHZI0OyRAAghI0AghYRRxBlEHZYmO84yIjA4PCqMw4jK4gKPOgzquuKIyqLgOihuIDgqyiyCrIKtBQgSykH1PzvPHqaIrnapea+uuz/v1Oq97695bt0/ddLq/fe4950RKCUmSJA1tbY2ugCRJkmrP0CdJktQCDH2SJEktwNAnSZLUAgx9kiRJLcDQJ0mS1AIMfZIkSS3A0CdJktQCDH2SJEktwNAnSZLUAoY3ugLNaOrUqWn69OmNroYkSVKP7rjjjqUppWk9HWfoK2P69Oncfvvtja6GJElSjyLi8d4c5+1dSZKkFmDokyRJagGGPkmSpBbgM32SJGlQ27x5M4sWLWLDhg2NrkpNjR49mj322IMRI0b06/2GPkmSNKgtWrSICRMmMH36dCKi0dWpiZQSy5YtY9GiRcyYMaNf5/D2riRJGtQ2bNjAlClThmzgA4gIpkyZMqDWTEOfJEka9IZy4Csa6Gc09EmSJA3AihUr+OIXv9jn973yla9kxYoVNahReYY+SZKkAagU+rZu3drt+6666ira29trVa0d2JFDkiRpAN773vfy6KOPMnv2bEaMGMH48ePZddddueuuu7j//vs54YQTeOKJJ9iwYQPveMc7OP3004HOGcDWrFnDK17xCo466ihuvvlmdt99d37yk58wZsyYqtbT0CdJkoaOs8+Gu+6q7jlnz4bPfKbi7gsvvJD77ruPu+66i+uuu45XvepV3Hfffc/1sr3kkkvYaaedWL9+PS960Ys4+eSTmTJlynbnePjhh/nud7/LV7/6VV73utfxwx/+kDe84Q1V/RhNcXs3IhZExEURcUNErIqIFBGXVeG8pxTOlSLiLdWoa1U8/TRceSWsXt3omkiSpCo7/PDDtxtW5XOf+xyHHnooc+fO5YknnuDhhx/e4T0zZsxg9uzZALzwhS9k4cKFVa9Xs7T0vR84FFgDLAIOGOgJI2JP4KLCOccP9HxVdcstcOKJ8Ic/wGGHNbo2kiQNHd20yNXLuHHjnlu/7rrruOaaa7jlllsYO3YsRx99dNlhV0aNGvXc+rBhw1i/fn3V69UULX3AOcAsYCLw1oGeLHKf5v8GlgFfHuj5qq740GYde+xIkqTamDBhAqsr3L1buXIlkydPZuzYsfzpT3/i1ltvrXPtOjVFS19K6driepXG2Xk7cAxwdGHZXIqhb+XKxtZDkiQN2JQpU5g/fz4HH3wwY8aMYeedd35u33HHHceXv/xlnv/857P//vszd+7chtWzKUJfNUXEgcCFwGdTStdHRPOGPlv6JEkaEr7zne+U3T5q1Ch+8YtflN1XfG5v6tSp3Hfffc9tP/fcc6teP2ie27tVERHDgW8BfwHOa3B1Kps0KS8NfZIkqU6GWkvfB4HDgKNSStV/ArJaJk7MS0OfJEmqkyHT0hcRh5Nb9z6ZUrqlH+8/PSJuj4jblyxZUv0Klho2LAc/Q58kSaqTIRH6Sm7rPgR8oD/nSCldnFKak1KaM23atKrWr6z2djtySJKkuhkSoY88Dt8s4EBgQ8mAzAk4v3DMVwvbGj+AD+TQZ0ufJEmqk6HyTN9G4OsV9r2A/JzfjcCDQJ9v/dbEpEmGPkmSVDeDLvRFxAhgX2BzSulRgEKnjbLTrEXEBeTQ942U0tfqVc8etbfDE080uhaSJKnOxo8fz5o1a+r+dZsi9EXECcAJhZe7FJZHRsSlhfWlKaXioDW7Aw8AjwPT61XHqmtvh3vvbXQtJElSi2iK0AfMBk7tsm2fQoEc8GozUmGj+EyfJElDwnve8x723ntv3va2twFwwQUXEBFcf/31PPvss2zevJmPfOQjvOY1r2loPZsi9KWULgAu6OWxC4Fez9XWl3PXVXs7rFoF27ZB21DpTyNJUmOdfTbcdVd1zzl7Nnymm26gr3/96zn77LOfC30/+MEPuPrqqznnnHOYOHEiS5cuZe7cubz61a+u1nSz/dIUoa8ltbfnwLdmTedgzZIkadA57LDDeOaZZ1i8eDFLlixh8uTJ7Lrrrpxzzjlcf/31tLW18eSTT/L000+zyy679HzCGjH0NUrpVGyGPkmSqqK7FrlaWrBgAZdffjlPPfUUr3/96/n2t7/NkiVLuOOOOxgxYgTTp09nw4YNjalcgfcVG6W9PS99rk+SpEHv9a9/Pd/73ve4/PLLWbBgAStXrqSjo4MRI0Zw7bXX8vjjjze6irb0NYyhT5KkIeOggw5i9erV7L777uy666780z/9E8cffzxz5sxh9uzZHHDAAY2uoqGvYYqhz6nYJEkaEu4tGYpt6tSp3HJL+fkgGjFGH3h7t3Fs6ZMkSXVk6GuU0o4ckiRJNWboaxRDnyRJqiNDX6OMGAHjxhn6JEmqgpRSo6tQcwP9jIa+RnIqNkmSBmz06NEsW7ZsSAe/lBLLli1j9OjR/T6HvXcbqb3d3ruSJA3QHnvswaJFi1iyZEmjq1JTo0ePZo899uj3+w19jWRLnyRJAzZixAhmzJjR6Go0PW/vNtKkSYY+SZJUF4a+RrKlT5Ik1Ymhr5EMfZIkqU4MfY1U7MgxhHsbSZKk5mDoa6T2dtiyBdata3RNJEnSEGfoayRn5ZAkSXVi6Guk9va8NPRJkqQaM/Q1kqFPkiTViaGvkQx9kiSpTgx9jVQMfU7FJkmSaszQ10i29EmSpDox9DWSvXclSVKdGPoaadQoGD3a0CdJkmrO0NdoTsUmSZLqwNDXaMWp2CRJkmrI0NdotvRJkqQ6MPQ12qRJhj5JklRzhr5Gs6VPkiTVgaGv0Qx9kiSpDgx9jWbokyRJddAUoS8iFkTERRFxQ0SsiogUEZf14zwfi4hfR8QTEbE+IpZHxJ0RcX5ETKlF3QesvR02bYINGxpdE0mSNIQ1RegD3g+cBcwGnhzAec4BxgH/C3wW+DawBbgAuCci9hxYNWvAqdgkSVIdDG90BQrOARYBjwAvAa7t53kmppR2aDKLiI8C5wH/Drytv5WsidKp2HbZpbF1kSRJQ1ZTtPSllK5NKT2cUkoDPE+le6Q/KCxnDuT8NWFLnyRJqoOmCH11cHxheU9Da1GOoU+SJNVBs9zeraqIOBcYD0wC5gBHkQPfhY2sV1nF0OdUbJIkqYaGZOgDzgV2Lnl9NfCmlNKSBtWnMlv6JElSHQzJ27sppV1SSgHsApwE7APcGREvqPSeiDg9Im6PiNuXLKljNiztyCFJklQjQzL0FaWUnk4pXQEcC0wBvtnNsRenlOaklOZMmzatbnVkzBgYMcLQJ0mSampIh76ilNLjwP3AQRExtdH12U6Es3JIkqSaa4nQV7BbYbm1obUox9AnSZJqbNCFvogYEREHRMS+XbYfEBE7jG4cEW2FwZk7gJtTSs/Wq6691t5u711JklRTTdF7NyJOAE4ovCwGtyMj4tLC+tKU0rmF9d2BB4DHgeklpzkO+EREXA88Ciwj9+B9Cbkjx1PAv9ToIwyMLX2SJKnGmiL0kefcPbXLtn0KBXLAO5fuXQNcDMwHDgXagbXAQ8C3gM+llJZXq8JVNWkSLFrU6FpIkqQhrClCX0rpAuCCXh67EIgy2+8DzqxmverGlj5JklRjg+6ZviHJ0CdJkmrM0NcM2tth/XrYtKnRNZEkSUOUoa8ZOP+uJEmqMUNfM3D+XUmSVGOGvmbg/LuSJKnGDH3NwJY+SZJUY4a+ZmDokyRJNWboawZ25JAkSTVm6GsGtvRJkqQaM/Q1g3HjYNgwQ58kSaoZQ18ziMg9eA19kiSpRgx9zcKp2CRJUg0Z+pqFoU+SJNWQoa9ZtLfbe1eSJNWMoa9Z2NInSZJqyNDXLOzIIUmSasjQ1yxs6ZMkSTVk6GsW7e2wZg1s2dLomkiSpCHI0NcsirNyrFrV2HpIkqQhydDXLJyKTZIk1ZChr1lMmpSXhj5JklQDhr5mYUufJEmqIUNfszD0SZKkGjL0NQtDnyRJqiFDX7Mohj6nYpMkSTVg6GsWEyZAhC19kiSpJgx9zaKtDSZONPRJkqSaMPQ1wHXXweGHw2OPddnhVGySJKlGDH0NsHkz/P73sHhxlx2GPkmSVCOGvgbo6MjLZ57psqO93Y4ckiSpJgx9DTBtWl4uWdJlhy19kiSpRgx9DTB1al7u0NI3aZKhT5Ik1URThL6IWBARF0XEDRGxKiJSRFzWx3NMiYi3RMQVEfFIRKyPiJURcWNEvDkimuKzAowcCZMnV7i9a+iTJEk1MLzRFSh4P3AosAZYBBzQj3O8FvgS8FfgWuAvwM7AScDXgFdExGtTSqkqNR6gjo4KoW/VKti2LQ/hIkmSVCXNEvrOIYe9R4CXkENbXz0EvBq4MqW0rbgxIs4DbgNOJgfAHw64tlVQMfSllINfcYYOSZKkKmiK5qSU0rUppYcH0gqXUvpNSulnpYGvsP0p4MuFl0cPoJpVNW1ahdAH9uCVJElV1xShrw42F5ZbGlqLEh0dFXrvgs/1SZKkqhvyoS8ihgNvLLy8upF1KdXRAUuXwtatJRsnTcpLQ58kSaqyIR/6gAuBg4GrUkq/bHRlijo68uN7y5aVbLSlT5Ik1ciQDn0R8XbgncCfgFN6OPb0iLg9Im5fssN91+orOyuHoU+SJNXIkA19EXEm8FngfuClKaXl3R2fUro4pTQnpTRnWnHKjBoqfomyoc+OHJIkqcqGZOiLiLOBzwP3kQPfUw2u0g6KLX3bNSpOnJiXtvRJkqQqG3KhLyLeA3wauIsc+LoOjNIUyt7eHT4cxo839EmSpKobdKEvIkZExAERsW+ZfR8gd9y4A3hZSmlp3SvYSzvtlCfdcCo2SZJUD00xI0dEnACcUHi5S2F5ZERcWlhfmlI6t7C+O/AA8DgwveQcpwL/AWwFbgDeHhFdv9TClNKlXTc2QltbNwM0G/okSVKVNUXoA2YDp3bZtk+hQA5459K9GYXlMODsCsf8Fri0H/WrCUOfJEmql6a4vZtSuiClFN2U6SXHLuy6rZfniJTS0XX+aN2qOCuHvXclSVKVNUXoa1UdHbb0SZKk+jD0NVDZ0DdpkqFPkiRVnaGvgTo68p3cjRtLNhZb+lJqWL0kSdLQY+hroOKsHNs919feDtu2wZo1DamTJEkamgx9DVR2Vg6nYpMkSTVg6GugsrNyFEOfz/VJkqQqMvQ1UNnQN2lSXhr6JElSFRn6GsiWPkmSVC+GvgaaMAFGjjT0SZKk2jP0NVBEmbH6DH2SJKkGDH0NtsNUbMVn+uy9K0mSqsjQ12A7tPSNHAljx9rSJ0mSqsrQ12BOxSZJkurB0Ndg06bl0LfdrGvFqdgkSZKqxNDXYB0dsH49rF1bstHQJ0mSqszQ12AVp2KzI4ckSaoiQ1+DVRyg2ZY+SZJURYa+Bqs4FZuhT5IkVZGhr8GmTcvLsi192/XukCRJ6j9DX4NVDH2bN+ceHpIkSVVg6GuwsWNh/PgyHTnAW7ySJKlqDH1NoOL8u/bglSRJVVLV0BcRkyNiXDXP2Qoqhj5b+iRJUpX0OfRFxMsi4uMRMblkW0dE/BZYCiyPiE9Vs5JDXXFWjudMmpSXhj5JklQl/Wnp+1fgpJTSsyXb/gv4G+ARYBnwjoh4XRXq1xJs6ZMkSbXWn9B3KHBj8UVEjAEWAP+bUtof2B94AjijKjVsAR0duSPHcyO0GPokSVKV9Sf0dQCLS14fAYwGLgVIKa0Gfk4Of+qFjg7YsqUk49mRQ5IkVVl/Qt9GYEzJ678BEnB9ybZVwE4DqFdL2WFWjtGjYdQoW/okSVLV9Cf0/Rk4puT1ycDDKaUnS7btSe7UoV4oO0CzU7FJkqQq6k/o+wZwSET8LiJuAA4BvtPlmBcADw60cq2i7Py7xanYJEmSqmB4P97zJWAu8A9AAD8DPlbcGRGHAwcC361GBVuBoU+SJNVan0NfSmkz8H8i4oz8Mq3ucshjwGHAwoFXrzVMnZqX203FNnkyLFvWkPpIkqShp98zcqSUVpUJfKSUlqaU7k4p9brraUQsiIiLIuKGiFgVESkiLutrnap1nnobMQJ22qlLS9+MGfDYYw2rkyRJGlr63NJXmIljV+DRlNLGku2nAScAa4HPpJRu68Np308e/28NsAg4oK/1qvJ56m6HWTlmzoTly3Nr35QpDauXJEkaGvrT0vefwO9K3xsR/wp8DTgeeD1wXUQ8rw/nPAeYBUwE3tqPOlX7PHW3w6wcs2bl5cMPN6Q+kiRpaOlP6JsP/DqltL5k27nAk8CLgeL0a//W2xOmlK5NKT2c0nNzUvRLtc7TCDuEvpkz89LQJ0mSqqA/oW938lh9ABRa9PYELkop3ZhSupzco/fF1aliayhOxfacGTOgrQ0eeqhhdZIkSUNHf0LfGGBDyev55Bk5rinZ9ig5HKqXOjry43tbthQ2jByZg58tfZIkqQr6E/qeZPsOEi8nT7t2d8m2yUDp7V/1YNo0SKnLKC0zZ9rSJ0mSqqI/oe9a4JURcVZEvAV4NXB1SmlbyTH7AU9Uo4L1EhGnR8TtEXH7ku3us9ZH2QGaZ87MLX2D7xFFSZLUZPoT+v4feUiUzwIXk2/1XlDcGREdwEuAm6tQv7pJKV2cUpqTUpozrTgZbh2VDX2zZsGaNfD003WvjyRJGlr6MyPHnyPiIGBBYdNPU0p/KTlkb+AL7Dgfr7pRsaUP8i3eXXape50kSdLQ0Z+5d0kpPQV8vsK+3wO/H0ilWlEx9G13Z7l0rL4X2xlakiT1X79CX1FEjCB36mgHVgIPFObmrZnC19wX2JxSerSWX6ueJk+GYcO6tPTttVfuxWtnDkmSNED9Cn0RMRH4OHAKMLpk14aI+Bbw3pTSij6c7wTyFG4AxfuYR0bEpYX1pSmlcwvruwMPAI8D0wdwnqbS1gZTp3YJfcOGwT77OGyLJEkasP7MvTsRuAk4CFgN3AD8lTwf72zgdOCoiJiXUlrVy9POBk7tsm2fQoEc8HoT1qp1nobYYVYOyLd4DX2SJGmA+tN799/Jge9LwN4ppaNTSv+YUjqazk4czysc1ysppQtSStFNmV5y7MKu2/pznmZUNvTNnAmPPALbtpV9jyRJUm/0J/SdBNyaUjqz6y3clNLKlNK/ArcAJ1ejgq1kh6nYILf0bdgAixY1pE6SJGlo6E/o2wu4rodjfkuej1d9MG1ahZY+sDOHJEkakP6EvnVARw/HTCscpz7o6IBVq3LD3nOKoc/n+iRJ0gD0J/T9HnhtRMwstzMi9gVeh2P19VnZsfp22w3GjjX0SZKkAelP6PsEMB74fUR8OCKOiYgDI+KlEfEhctgbD/xXNSvaCsrOytHWBvvt5+1dSZI0IP2Zhu3XEfE28ty75xVKUQCbgbNSStdUp4qto2xLH+RbvPfeW/f6SJKkoaO/07B9JSJ+QR6c+TBgEnlGjjuBy1JKj1eviq1j2rS8LDtW309+Alu2wPABTaIiSZJaVL8TRErpL8BHy+2LiNHAyD4Mziwq3N6F3NK3ZQssXJhv9UqSJPVRf57p640vActrdO4ha8IEGDWqQksf2JlDkiT1W61CH+Tn+9QHEd3MygF25pAkSf1Wy9Cnfigb+qZNg4kTbemTJEn9ZuhrMtOmlem9G5Fv8drSJ0mS+snQ12TKtvRBvsVrS58kSeonQ1+TKYa+lLrsmDULHn8cNm5sSL0kSdLgZuhrMh0dee7dNWu67Jg5MyfBRx9tSL0kSdLg1qtx+iJia60roqx0rL4JE0p2FHvwPvwwPO95da+XJEka3Hrb0hf9KOqHbqdiAztzSJKkfulVS19KydvAdVJxKrbJk2HqVDtzSJKkfjHMNZmKU7GBPXglSVK/GfqaTMWWPnCsPkmS1G+GviYzZkzuwFGxpW/x4jJdeyVJkrpn6GtCHR1lOnJAbukDeOSRutZHkiQNfoa+JjRtWjctfeBzfZIkqc8MfU2o4lRs++2Xl4Y+SZLUR4a+JlQx9I0fD7vtZmcOSZLUZ4a+JlR8pm/btjI7HbZFkiT1g6GvCXV0wNat8OyzZXY6bIskSeoHQ18TKo7VV7YH78yZsHQprFhR1zpJkqTBzdDXhHqclQO8xStJkvrE0NeEug19xbH6vMUrSZL6wNDXhLoNffvsAxG29EmSpD4x9DWhqVPzsmzoGz0a9t7b0CdJkvqk4aEvIhZExEURcUNErIqIFBGX9fNce0TEJRGxOCI2RsTCiPhMREyudr1rafhw2GmnCh05ID/X5+1dSZLUBw0PfcD7gbOA2cCT/T1JROwL3AGcBtwGfBp4DHgHcEtETBl4Veun4gDN0DlWX0p1rZMkSRq8miH0nQPMAiYCbx3Aeb4IdABvTymdkFJ6b0rpGHL42x/46IBrWkcdHfD00xV2zpoFK1d20xQoSZK0vYaHvpTStSmlh1Pqf7NVROwDHAssBL7QZff5wFrglIgY1++K1tn++8O993YzKwf4XJ8kSeq1hoe+KjmmsPxVSmm7mJRSWg3cBIwF5ta7Yv01b14ef/mBB8rsNPRJkqQ+Giqhb//CslLvhmI6mlWHulTF/Pl5edNNZXZOn557e9iZQ5Ik9dJQCX2TCsuVFfYXt7dXOkFEnB4Rt0fE7Uua4Fm5/fbL07HdfHOZnSNGwIwZtvRJkqReGyqhrydRWFZ8bjCldHFKaU5Kac604uS3DRSRb/GWbemD3JnDlj5JktRLQyX0FVvyJlXYP7HLcYPCvHnwyCPdzMH7yCMO2yJJknplqIS+BwvLSs/sFXo+VHzmrykVn+sre4t35kxYtw4WL65rnSRJ0uA0VELftYXlsRGx3WeKiAnAfGA9cGu9KzYQL3whjBxZIfTNKuRbb/FKkqReGFShLyJGRMQBhdk3npNSehT4FTAdOLPL2z4EjAO+mVJaW5eKVsno0Tn4lX2u78AD8/IPf6hrnSRJ0uA0vNEViIgTgBMKL3cpLI+MiEsL60tTSucW1ncHHgAeJwe8Um8DbgY+FxEvKxx3BPBS8m3d99Wi/rU2bx5cdBFs3AijRpXs2H13OPRQuOIKeOc7G1Y/SZI0ODRDS99s4NRCeXlh2z4l2xb05iSF1r45wKXksPdOYF/gc8CRKaVlVa11ncyfD5s2wR13lNl58sm5GdDn+iRJUg8aHvpSSheklKKbMr3k2IVdt3U51xMppdNSSrumlEamlPZOKb0jpbS8Xp+n2ubNy8uyz/UtKOThK66oW30kSdLg1PDQp+7tvDPsu283z/U973lw+eV1r5ckSRpcDH2DwLx5uaWv7JB8J58M119fYTA/SZKkzNA3CMyfnzPdo4+W2blgAWzbBj/+cd3rJUmSBg9D3yDQ7XN9hxySJ+r1Fq8kSeqGoW8QOOggmDixwnN9Ebm17ze/gWWDsoOyJEmqA0PfINDWBkceWaGlD3Lo27oVfvrTutZLkiQNHoa+QWL+fPjjH2HFijI7X/ACmD4dfvjDeldLkiQNEoa+QWL+/Nx799ZyswdH5F68v/oVrFxZ97pJkqTmZ+gbJA4/HIYNq/BcH+TQt3kz/Oxnda2XJEkaHAx9g8T48Xmq3YrP9R1xRJ6P11u8kiSpDEPfIDJvHvzud7BlS5mdbW1w0klw9dWwZk3d6yZJkpqboW8QmT8f1q6Fe+6pcMCCBbBhA1x1VV3rJUmSmp+hbxApDtJc8bm++fPzZL0O1CxJkrow9A0ie+0Fe+zRzXN9w4bBiSfmlr516+paN0mS1NwMfYPMvHndtPRBvsW7di388pd1q5MkSWp+hr5BZv58eOKJXMp6yUtgyhRv8UqSpO0Y+gaZ4nN9FW/xDh8OJ5yQx+vbuLFu9ZIkSc3N0DfIHHoojB3bi1u8q1fD//5v3eolSZKam6FvkBkxIs/OUbGlD+CYY2DSJAdqliRJzzH0DULz58Ndd3UzBvPIkfCa18CPfwybNtW1bpIkqTkZ+gahefNg61b4/e+7Oejkk2HFCrj22rrVS5IkNS9D3yB05JF52e1zfccemyfs9RavJEnC0DcoTZ4Mz3teD8/1jR4Nxx8PP/pRHrdPkiS1NEPfIDV/PtxyC2zb1s1BZ50Fy5bBJz5Rt3pJkqTmZOgbpObNy4/sPfBADwe97nXw8Y/DokV1q5skSWo+hr5B6qij8rLH2dYuvDD3+njf+2peJ0mS1LwMfYPUfvvlDh1f/nIPt3hnzIBzzoFvfhNuv71u9ZMkSc3F0DeInXkmPPxwLybeOO88mDYN/u3fIKW61E2SJDUXQ98gtmABdHTAF77Qw4ETJ8KHPww33JB780qSpJZj6BvERo2Cf/kX+PnP4c9/7uHgN78ZDj4Y3v1u2LixLvWTJEnNw9A3yP3f/wttbfnZvm4NHw6f+hQ89hhcdFFd6iZJkpqHoW+Q23PPPM3u174G69f3cPDf/R288pX5Vu+SJXWpnyRJag5NE/oiYo+IuCQiFkfExohYGBGfiYjJfTzPiRHxm4hYEREbIuKBiPhgRIyuVd0b7ayzYPly+P73e3Hwf/1XnqHj/PNrXi9JktQ8miL0RcS+wB3AacBtwKeBx4B3ALdExJRenufDwI+AFwE/Br4ArAI+BFwTEWOqX/vGO/roPC3b5z/fi865Bx4Ib30rfOUr8Mc/1qN6kiSpCTRF6AO+CHQAb08pnZBSem9K6Rhy+Nsf+GhPJ4iIw4D3ASuAQ1NKb0opvROYC3wemA+8p1YfoJEi8vAtd9wBt93Wizecfz5MmADnnlvzukmSpObQ8NAXEfsAxwILyS1zpc4H1gKnRMS4Hk51IhDA11JKjxU3ppQScB6QgLdGxLAqVb2pnHJKznGf/3wvDp46FT74Qbj66lwkSdKQ1/DQBxxTWP4qpbTd3BIppdXATcBYcotdd3YpLB/ruqNwnqXk1sRDBlTbJjVhApx6KvzgB/DMM714w1ln5Wk93vlO2LKl5vWTJEmN1Qyhb//C8qEK+x8uLGf1cJ6lheU6pT4hAAAgAElEQVSMrjsiYgIwtfDygD7VbhA580zYtCn35O3RyJHw8Y/D/ffD5z5X87pJkqTGaobQN6mwXFlhf3F7ew/n+Xlh+ZaImN5l30fIt34ByvYGjojTI+L2iLh9ySAdzuSAA+BlL8tj9vWq8e6EE+D44+Fd74Kf/azm9ZMkSY3TDKGvJ8Ww1m2/1JTSzcBXyKHunoj474j4ZETcCpwFFLuqbq3w/otTSnNSSnOmTZtWparX31lnwRNP9DLDRcB3vgMveAH8wz/ArbfWvH6SJKkxmiH0FVvyJlXYP7HLcRWllM4A3gzcD7wOOAPYBLwcuLdwWG+eeBu0/v7v84DNPc7HWzR+PFx5Jey2W37zgw/WtH6SJKkxmiH0FVNGpWf2ZhaWlZ75205K6ZKU0tyU0rhCeXFK6RrgyMIhvx9AXZve8OF5GL5f/xoeeKCXb+royL1429rguOPgqadqWkdJklR/zRD6ri0sj42I7epT6IAxH1gP9PveY0QcC+wN/Dal9GR/zzNYvOUtuZ/GF7/Yhzftt19u8XvmmTxV26pVNaufJEmqv4aHvpTSo8CvgOnAmV12fwgYB3wzpbS2uDEiDoiIHXrhRsTEMtv2BS4mP8v33urVvHlNm5Yf0bv00j5mtxe9CC6/HO65B04+OXcFliRJQ0LDQ1/B28jP2n0uIn4cEf8vIn4DnEO+rfu+Lsc/UChdfT0ibouIL0XEf0bED4D7gD2At6SUWqanwplnwpo18K1v9fGNr3hFHvPlmmvgn/8Ztm3r+T2SJKnpNUXoK7T2zQEuBY4A3gnsC3wOODKltKyXp/o5sJncieNcYB7wQ+AFKaVLq1vr5nb44TBnTp6ho89jL7/pTfCRj8C3vw3//u+1qJ4kSaqz4Y2uQFFK6QngtF4eGxW2fwP4RjXrNVhFwHnnwUknwSc+0Y/sdt55sGhRHsB5993h7W+vST0lSVJ9NEVLn2rjxBPhda+D88+He+/t+fjtRORmwhNOgLPPhne/GzZurEk9JUlS7Rn6hrgvfAEmT853bDdv7uObhw3LgzeffnpuLjziCPjjH3t+nyRJajqGviFu6tQ8Ldsf/gAXXtiPE4wZk0/w05/C4sXwwhfmuXrt4CFJ0qBi6GsBJ54I//iP8B//AXfd1c+THH98vkf8t38L73hHHsR58eKq1lOSJNWOoa9FXHQRTJmSb/P2e/i9nXfOk/p+6Utw441wyCF5XD9JktT0DH0tYsoU+MpX4O674aMfHcCJIuCMM+DOO2GffeC1r4XTTnMGD0mSmpyhr4W85jVwyik59P3hDwM82f77w803wwc+AN/8JjzvebnTR0pVqaskSaouQ1+L+exnoaMDTj21CiOwjBiRHxS86aZ86/ef/gmOOgpuv70qdZUkSdVj6GsxkyfDV78K992X81pVzJ0Lv/89fP3r8MgjeQ7ff/5neOqpKn0BSZI0UIa+FvSqV+XH8C68MGe1qmhry0Hv4YfhXe+Cyy6DmTPzjB4O6ixJUsMZ+lrUpz4Fu+2Wb/Nu2FDFE0+cmIPeH/8IxxwD73kPHHQQ/OQnPu8nSVIDGfpaVHs7fO1r8MADcOaZNRhreebMHPR++UsYNSpP53bwwfCZz8CyZVX+YpIkqSeGvhb28pfD+98Pl1ySZ1qrySQbxx6bR4T++tdh/Hg45xzYfffc6eO662z9kySpTgx9Le4//iOPuvL1r8Ob3wxbt9bgi4wYkZ/3+93v8kCB//IvcOWV8NKX5qFfPv5xeOaZGnxhSZJUZOhrcRE5+H3oQ3DppXnGjpoEv6LnPz9PD7J4MXzjG3mol/e8J7f+vfrV8OlPwx13wJYtNayEJEmtZ3ijK6Dm8MEPwrBh+Xbvli3wrW/B8Fp+d4wdC298Yy4PPJAfMLziijzNG+RbwfPmwd/8TS6HHw5jxtSwQpIkDW2RfKZqB3PmzEm3t+gAwx/7GLz3vXl2tW9/O9+Zrasnn4Qbbugs996bt48cmcf/+9u/hVe+EubMycPESJLU4iLijpTSnB6PM/TtqJVDH8AnPwnnngsnngjf+17OWw2zfHme8eOGG+D66/PAgtu25WlFXvGKHACPPTZ3R5YkqQUZ+gag1UMf5Onazj47P2b3gx/kUVeawrJlcPXVcNVVebl8eb4vPX9+HnX65S+HAw5oogpLklRbhr4BMPRlX/gCnHVWzlLf/S5MmNDoGnWxdWvuEXzllbncfXfe3tYG06fDrFl5vMBZszrLnnvmkChJ0hBh6BsAQ1+nr3wF3vpW2Htv+O//hqOPbnSNuvHkk3nsvwcfzNPBPfRQLmvWdB4zcmRuCZw9Gw47LJfZs2HSpIZVW5KkgTD0DYChb3s335yna3vkEXjHO+A//zN3vh0UUoKnnuoMgQ8+CPfdB3feCU8/3XncPvt0hsBDD80pd5ddYMoUO4xIkpqaoW8ADH07WrsW/v3f8xB7s2blIfbmzm10rQboqady+Cstjz66/THDhuWxBHfZBXbdNS+LZY89ctl993yM4VCS1ACGvgEw9FX2m9/AaafBokV5TOXzzx9ifSZWrcotgU8+mUNhufL00zuOYD18OOy2Ww6AxTC4//651fCQQ2DcuMZ8HknSkGfoGwBDX/dWrYJ/+7c8ddshh+RWv8MOa3St6mjbNliyJAfDRYty6br+xBOwbl0+PgL22y8HwNKy5555nyRJA2DoGwBDX+9ceWWeRnfJEnjXu/LYfjvt1OhaNYmU4PHHc4/i0lJ6+3js2NxM2taWw19b2/brEXl07FGjchk9unO9+Hr06HxrudjKWCy77pr3SZKGPEPfABj6em/58ty547LL8pAub387nHNO7v+gMlatyrOM3H137lyydWtuOdy2LQfF0vWtW/OceBs35rJhw47r69bl280bNuz4taZMyQFw8mTYvHn7smnT9q/Hj4epU/N7SpfF9WnTOoPloOnFI0mtwdA3AIa+vrvnHvjwh+Hyy3N+OOsseOc7c2ZQjaUEzz6bbysvXpyXxbJ4MaxYkVsMR4zIQ9aULotlzRpYujQPfl1crl5d/utNmrR9q2IxDE6YkENq11IMr1u35mcbJ02CiRPzsnR94sT8bOSKFbk8+2znsri+YkX+OnvuuX2xVVNSCzP0DYChr//uuw8+8pE8i8fYsfC2t+Xbvh0dja6Z+mzjxhz+li2DZ57ZMVAWQ+Vf/7pjx5ZaGTkyt1J2NW1aZwDs6Mgtn2vXVi7btuVzlYbgruvjxuWAOX789qW4bdSo7UNtubJpU65LpbJ5M4wZk/+zjBtXfgm5zuvW5VJcLy43bMiBefLk/HzFTjvtuD56dD5+zZryZe3a/PV23jlfv5137lwfP77ys6ebN29/XTdtyn+EFEux1bp0vdhSXWytLr0eGzfm40qvQWkpbhsxovNxiHIlorPVvLQ1vS/bSl9X2lf8niz3B9XIkfmPmLVr8y2R4h8vxfXicu3azsc1xowpvxw/Pv9h1N7e+cdSe3vlf5uu13n9+vxH3PLlnf+nu5Znn83Xt9jCX65MnpzrVKzv8OG9/7+bUv5+2bo1v3ewPs+8dWu+Y1MsxbssETuW4vZdd635L0FD3wAY+gbu/vtz+Pve9/LPiLe+Fc48E2bMaHTNVHVbt+ZQuHZt/mU3fPj2ZdiwvGxry8esXJl/WK5cuf36qlX5l0J7e/7lUlyWro8enX+BLVqUO8uUK0uW5G+6roGhtLS1dd7mLt7qLq5v2tR563zNmvzLsrjcsqXv16f0+cuuZdiw/Hm6Brpyt+shv6drKBw9uvMX+vLl+Xz9qePGjeX3jRmTf2FNnpzPXQyPa9fm66b+GTMmX9Px4zsD2vr1eVnuD5ty2tpy4J8wIf9blAbo3tZhypRciv++S5fmsnJl775+8Vnj4hLK/98q/b8zbFhncC0G2eJ6e3v+eVEarErLypWddyFK71Z0vYMxcuT2f6yNG7fja8ifufh/sLheum316s6fT6tW5e/7vvrYx+Dd7+77+/rA0DcAhr7q+dOf4KMfhe98J/9hfPTR8KY3wckn5/930qCyaVNnANy4Mf9yKYbackG32CLVV1u3dgasiBzuxo7t3bk2bOhsSSqWjRt3bLEs/eU3bFj+bEuW5GdEn3kml+L600/nW+tdw3Tx/cUycmRnK0exxa3r664huPT1qFH5mGIALm3VLC3lWuq6lmHDOlv+Stf7sq3SOYrrxdar0nDTdTl+fOcfL8WW1+IfMN39+2/c2Bk+1qzp/COpWFas6FxfvTp/r5W2DhZL8fW4cZ0Br1jGjOn+e3358s4QuHRp/r4qbaHtutywobMDWrmWz5Ej83Vbs6bzcY3iZyktW7bkMFv66EdpKc4J2t2zyhs3bt+6Xbre9Q+ctrZ8LbqWsWO3/7rl6lL8dyxmqdKW7uLrgw/OA9zWkKFvAAx91feXv8C3vgWXXppn9hg3DhYsyAHwxS92XGNJUp0UH0uAHOxGjBi8t5sLehv6muZXbUTsERGXRMTiiNgYEQsj4jMRMbmP5zkqIn5SeP+GiPhLRFwVEcfVqu7q2V57wfvel2dCu/FG+Md/hB/9CF76Uth33zzI8yOPNLqWkqQhb8SIzlvJxdbpFtEULX0RsS9wM9AB/AT4E3A48FLgQWB+SmlZL87zVuCLwFrgCmARsAdwEjAWeH9K6aM9nceWvvpYtw6uuCIP7nzNNbkV/MAD4e//Hl71Kpg3L//flCRJlQ2q27sR8UvgWODtKaWLSrZ/CjgH+EpK6YwezjECWAKMAmanlB4s2XcgcCewDZicUqrwxHJm6Ku/J56AH/4wD/j82992Ps//8pfnEHjccQ7/IklSOYMm9EXEPsCjwEJg35TStpJ9E4C/AgF0pJQqdpuJiJ2Bp4B7UkqHltl/D3AIMLWnVkNDX2OtWpVb/n7+8xwCn3kmP/M3dy687GW5BXDu3BwKJUlqdb0NfX0YZKdmjiksf1Ua+ABSSqsj4iZyK+Bc4NfdnOcZckvfrIiYmVJ6uLgjImYBM4G7enObWI01cSKcdFIu27bBHXfkAHjVVbkn8LZt+RGMgw7KAXD+/Lzcd9+WejRDkqQ+aYbQt39h+VCF/Q+TQ98sugl9KaUUEWcClwF3RMQVwGJgd+BE4I/A66tVadVHWxu86EW5fOhDeWSC226Dm2/O5fvfh4svzsd2dOQWwMMOg+c/P5d99rFnsCRJ0Byhb1JhWWkkyOL2Hm/mpZT+JyIWA98F3liy62ngv4HHKr03Ik4HTgfYa6+9evpSapAJE/It3pe9LL/eti0PBF0MgbfemlsFi4PljxsHhxySA+Chh+blQQflYbIkSWolzRD6elK8Ydfjw4cR8Qbgq8CPgA8DjwN7Ax8APg+8BHhdufemlC4GLob8TN+Aa626aGvL414efDCcfnretm5dDoJ3353nBL77bvif/+lsEYQ8a9esWbD//rkU1/fZp3NQeUmShpJmCH3FlrxJFfZP7HJcWYXn9i4B7gFOKXk+8E8RcQr5NvJrI+LolNJ1A6uymtnYsTBnTi5FKeWpYu++Gx54AB58MJcrr4RLLuk8rq0tTxU3YwZMn7592XvvPIXisGH1/TySJFVDM4S+4tAqleYomVlYVnrmr+hYYATw2zIdQrZFxPXACwvluv5VVYNVBOyxRy6vetX2+1asyINGP/RQDoIPPQQLF8JPf5p7DpcaMSIPNL3XXrD77rDbbp3L4vquu+bxPiVJaibNEPquLSyPjYi2MkO2zAfWA7f2cJ7iTblpFfYXt/dyJmu1ivZ2OPzwXLpaty5PIbdwITz+eF4uXJi33XgjLF5cfm70qVNhl11g551zB5OOjh3Xp03L019OmGCvY0lS7TU89KWUHo2IX5Fb6s4ELirZ/SFgHHlw5ufG6IuIAwrv/VPJsTcUlgsi4r9SSveUHD8bWEB+LvA3NfkgGpLGjoUDDsilnJRg2bJ863jx4lyefDKX4jz1t92W11evLn+OYcPyPOzlyuTJeY7vSZNyOC2ul7521hJJUm80fHBmKDsN2wPAEeRp2B4C5pWOrxcRCSClFF3OcwlwGrk17wpyR47pwAnASOAzKaVzeqqPgzOrFtavhyVLchB85plcnn02h8bly3csy5ZVDoqlRo3KrYXFMn78jq/HjculdL20jB2by5gxuYwdC6NH2wIpSYPBYBqcudjaNwf4D+A44JXkmTg+B3wopbS8l6d6M3A98Cbg5cAEYBVwI/DVlNL3qlx1qdfGjOl8HrC3tm7NM5SsWAErV3aW0terV+9Yli/Pt6CLr9euhS1b+l7n0aM7w+Do0d0vR43qXRk5svsyYkRn6fp6xAiDqCT1V1O09DUbW/o0FG3alMNfsaxZ07m+bl1uiVy/fvv10tcbNnQuK61v3NhZtm3ruU79MWxYZwAcPrzy+rBheVmuDBvWub/reum2rqXcvra23i+7W+9tieh+vbtlT9u6rpd7XakUj4Md9xW3SaqNQdXSJ6n2ii1p9RqYesuW7UPghg2weXMOn5XKxo35mGLZtGnH11u25PVKy82bcwvpli07lnXrOvcXy5YtO66Xbit3rH8r91/XIFgpIPZ2W7lz9mZfaQjt7r3dLSut97S/r+vlAnPXbT297u97ejpHb/bX6+v0VTXq0RtnnglveMPAz1MNhj5JNVFsVRs3rtE1qb6Ucvjbtq38slhKjys9pvS4lDq3dS3F47seU+l1pWVxvVgqHdt1f6XX5UqxZbfr9t5u68uxxW2l+7pu625faWjv7r3dLSut97S/r+vl/sDouq2n1/19T0/n6M3+en2dvqpGPXqrmYbwMvRJUh9F5EArSYOJU9FLkiS1AEOfJElSCzD0SZIktQBDnyRJUgsw9EmSJLUAQ58kSVILMPRJkiS1AEOfJElSCzD0SZIktQBDnyRJUgsw9EmSJLUAQ58kSVILMPRJkiS1gEgpNboOTScilgCP1/jLTAWW1vhrtCKva214XWvD61obXtfa8LrWRjWu694ppWk9HWToa5CIuD2lNKfR9RhqvK614XWtDa9rbXhda8PrWhv1vK7e3pUkSWoBhj5JkqQWYOhrnIsbXYEhyutaG17X2vC61obXtTa8rrVRt+vqM32SJEktwJY+SZKkFmDokyRJagGGvjqKiD0i4pKIWBwRGyNiYUR8JiImN7puzS4iFkTERRFxQ0SsiogUEZf18J55EXFVRCyPiHURcU9EnB0Rw+pV72YWEVMi4i0RcUVEPBIR6yNiZUTcGBFvjoiyPx+8rj2LiI9FxK8j4onCdV0eEXdGxPkRMaXCe7yu/RARpxR+HqSIeEuFY/4+Iq4rfH+viYjfRcSp9a5rsyr8LkoVylMV3uP3ay9FxN9ExA8j4q+F3/1/jYhfRcQryxxb0+vqM311EhH7AjcDHcBPgD8BhwMvBR4E5qeUljWuhs0tIu4CDgXWAIuAA4Bvp5TeUOH41wA/BDYA3weWA8cD+wOXp5ReW496N7OIOAP4EvBX4FrgL8DOwEnAJPL1e20q+SHhde2diNgE/AG4H3gGGAfMBeYAi4G5KaUnSo73uvZDROwJ3AsMA8YD/5JS+lqXY84CLgKWka/tJmABsAfwyZTSuXWtdBOKiIVAO/CZMrvXpJT+q8vxfr/2UkS8H/gwefDln5N/3k4FDgOuTSm9u+TY2l/XlJKlDgX4JZCAf+2y/VOF7V9udB2buZDD8UwggKML1+yyCsdOJP+i3QjMKdk+mhy8E/D6Rn+mRhfgmMIPlLYu23chB8AEnOx17de1HV1h+0cL1+mLXtcBX+MArgEeBT5RuE5v6XLMdPIv0GXA9JLtk4FHCu85stGfpdEFWAgs7OWxfr/2/rq+tnA9/heYUGb/iHpfV2/v1kFE7AMcS/6P9YUuu88H1gKnRMS4Oldt0EgpXZtSejgV/hf0YAEwDfheSun2knNsAN5fePnWGlRzUEkp/Sal9LOU0rYu258Cvlx4eXTJLq9rLxWuSTk/KCxnlmzzuvbP28l/uJxG/hlazj8Do4DPp5QWFjemlJ4F/rPw8owa1nEo8vu1FwqPx3wMWAf8n5TS6q7HpJQ2l7ysy3UdPtATqFeOKSx/VeYX7OqIuIkcCucCv6535Yag4vW+usy+68n/CedFxKiU0sb6VWtQKf4w2lKyzes6cMcXlveUbPO69lFEHAhcCHw2pXR9RBxT4dDuru0vuhzT6kZFxBuAvcgh+h7g+pTS1i7H+f3aO/OAGcDlwLMR8SrgYHLL820ppVu6HF+X62roq4/9C8uHKux/mBz6ZmHoq4aK1zultCUi/gwcBOwDPFDPig0GETEceGPhZekPIK9rH0XEueRnzSaRn+c7ivzL9MKSw7yufVD4/vwW+RGE83o4vLtr+9eIWAvsERFjU0rrqlvTQWcX8nUt9eeIOC2l9NuSbX6/9s6LCsunyc/3HlK6MyKuBxaklJYUNtXlunp7tz4mFZYrK+wvbm+vQ11agdd7YC4k/0V6VUrplyXbva59dy75EY6zyYHvauDYkh/04HXtqw+SH4J/U0ppfQ/H9vbaTqqwv1X8N/AycvAbRw4oXyE/E/mLiDi05Fi/X3uno7A8AxgD/C0wgfyz9ZfAi4H/KTm+LtfV0NccorC0K3V9eL0riIi3A+8k9y4/pa9vLyy9rgUppV1SSkH+ZXoS+a/0OyPiBX04jde1ICIOJ7fufbLM7bF+nbKwbOlrm1L6UOEZ36dTSutSSvellM4gdzQcA1zQh9N5TbPiECtBbtH7dUppTUrpj8CJ5FEoXhIRR/byfFW5roa++ujpr8mJXY7TwHi9+yEizgQ+Sx5m5KUppeVdDvG69lPhl+kV5Mc4pgDfLNntde2Fktu6DwEf6OXbenttVw2gakNZsUPXi0u2+f3aO88Wlo+llO4u3VFooS7eRTm8sKzLdTX01ceDheWsCvuLPfkqPfOnvql4vQu/OGaQOyg8Vs9KNbOIOBv4PHAfOfCVG5DV6zpAKaXHyaH6oIiYWtjsde2d8eRrdCCwoXQAYfItdICvFrYVx5vr7truSr6Vucjn+Sp6prAsHVnC79feKV6nFRX2F0PhmC7H1/S6Gvrq49rC8tiusxxExARgPrAeuLXeFRuiflNYHldm34uBscDNLd6z7DkR8R7g08Bd5MD3TIVDva7VsVthWewV6XXtnY3A1yuUOwvH3Fh4Xbz12921fUWXY7Sj4q3H0qDh92vvXE8OaTMjYmSZ/QcXlgsLy/pc10YPXtgqBQdnrua1PJqeB2degoOH9uZafqBwPW4HdurhWK9r767pAcAuZba30Tk4801e16pe8wsoPzjzDBycuadrd1C5//vA3uSRJRJwXsl2v197f20vK1yPj3TZ/nfANnIrYHs9r6vTsNVJmWnYHgCOIM808RAwLzkNW0URcQJwQuHlLsDLyX993lDYtjSVTKdUOP5y8g/875Gns3k1helsgNelFv/mL8w9eim5xekiyj8rsjCldGnJe7yuPSjcKv8E+S/9R8mBY2fgJeSOHE8BL0sp3V/yHq/rAETEBeRbvOWmYftX4HM4DVtZhWv3XvIdqT8Dq4F9gVeRA8dVwIkppU0l7/H7tRciogO4CdiP/LvqNnKYPpEc4v5PSul/So6v/XVtdBJupQLsSe4a/1fyD57HyQ/Od9vCYtnuL/lKZWGZ98wn/8B6lnz7/F7gHGBYoz9PM5ReXNMEXOd17fN1PZg8885d5Pk2t5AD9e8L17zs/3ev64CuefF7+S0V9h8P/JYcaNYW/i1ObXS9m6GQ/xj5LrnH/grywOxLyFOHvRFy41CZ9/n92rvruxP5jt6fC7/3l5EbfuY24rra0idJktQC7MghSZLUAgx9kiRJLcDQJ0mS1AIMfZIkSS3A0CdJktQCDH2SJEktwNAnSZLUAgx9kjQIRcQFEZEi4uhG10XS4GDok9SSCoGpp3J0o+spSdUyvNEVkKQG+1A3+xbWqxKSVGuGPkktLaV0QaPrIEn14O1dSeqF0mfoIuLUiLgzItZHxDMRcUlE7FLhfTMj4psR8WREbIqIxYXXMyscPywizoiImyJiZeFrPBIRX+vmPQsi4raIWBcRyyPiexGxezU/v6TBz5Y+Seqbc4Bjge8DVwNHAacBR0fEESmlJcUDI+JFwDXABOCnwP3AAcA/Aa+JiJellG4vOX4kcCXwt8ATwHeAVcB04ETgRuDhLvV5G/Dqwvl/CxwB/ANwaETMTiltrOaHlzR4GfoktbSIuKDCrg0ppQvLbH8FcERK6c6Sc3waOBu4EHhzYVsA3wQmAm9IKX275Ph/AL4HXBYRz0spbSvsuoAc+H4GvLY0sEXEqMK5ujoOeFFK6d6SY78D/CPwGuAHFT+8pJYSKaVG10GS6i4ievrhtzKl1F5y/AXA+cAlKaU3dznXJOBxYBTQnlLaGBHzyS1zt6SU5pX5+jeQWwlfklK6PiKGAcuAkcB+KaXFPdS/WJ+PppTe32XfS4HfAJ9MKZ3bw+eU1CJ8pk9SS0spRYXSXuEtvy1zjpXAXcBo4MDC5hcUlr+pcJ7i9sMKywOAScA9PQW+Lm4vs+2JwnJyH84jaYgz9ElS3zxdYftTheWkLsu/Vji+uL29y/LJPtZnRZltWwrLYX08l6QhzNAnSX2zc4Xtxd67K7ssy/bqBXbtclwxvNnrVlJNGPokqW9e0nVD4Zm+2cAG4IHC5mJHj6MrnKe4/Q+F5Z/Iwe/5EbFbNSoqSaUMfZLUN6dExGFdtl1Avp373ZIetzcBDwJHRcSC0oMLr18MPETu7EFKaSvwRWAM8OVCb93S94yMiGlV/iySWohDtkhqad0M2QLw45TSXV22/QK4KSJ+QH4u76hCWQi8t3hQSilFxKnA/wLfj4ifkFvz9gdOAFYDbywZrgXylHBHAMcDD0XEzwvH7UkeG/BdwKX9+qCSWp6hT1KrO7+bfQvJvXJLfRq4gjwu3zcXXxUAAACWSURBVD8Aa8hB7LyU0jOlB6aUflcYoPn95PH3jgeWAt8FPpxSerDL8Zsi4jjgDOCNwKlAAIsLX/PGvn88Scocp0+SeqFkXLyXppSua2xtJKnvfKZPkiSpBRj6JEmSWoChT5IkqQX4TJ8kSVILsKVPkiSpBRj6JEmSWoChT5IkqQUY+iRJklqAoU+SJKkFGPokSZJawP8HFCdEj2TItk4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(1, 1, figsize=(10,6))\n",
    "ax.plot(np.sqrt(exponential_decay_model_history.history['loss']), 'r', label='train')\n",
    "ax.plot(np.sqrt(exponential_decay_model_history.history['val_loss']), 'b' ,label='val')\n",
    "ax.set_xlabel(r'Epoch', fontsize=20)\n",
    "ax.set_ylabel(r'Loss', fontsize=20)\n",
    "ax.legend()\n",
    "ax.tick_params(labelsize=20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3 - Choosing an `optimizer` and a `loss function`\n",
    "\n",
    "When constructing a model and using it to make our predictions, for example to assign label scores to images (\"cat\", \"plane\", etc), we want to measure our success or failure by defining a \"loss\" function (or objective function). The goal of optimization is to efficiently calculate the parameters/weights that minimize this loss function. `keras` provides various types of [loss functions](https://github.com/keras-team/keras/blob/master/keras/losses.py).\n",
    "\n",
    "Sometimes the \"loss\" function measures the \"distance\". We can define this \"distance\" between two data points in various ways suitable to the problem or dataset.\n",
    "\n",
    "Distance \n",
    "\n",
    "- Euclidean \n",
    "- Manhattan\n",
    "- others such as Hamming which measures distances between strings, for example. The Hamming distance of \"carolin\" and \"cathrin\" is 3.\n",
    "\n",
    "Loss functions\n",
    "- MSE (for regression)\n",
    "- categorical cross-entropy (for classification)\n",
    "- binary cross entropy (for classification)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "# build the model\n",
    "input_dim = x_train.shape[1]\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(64, activation=tf.nn.relu, kernel_initializer='uniform', \n",
    "                input_dim = input_dim)) # fully-connected layer with 64 hidden units\n",
    "model.add(Dropout(0.1))\n",
    "model.add(Dense(64, kernel_initializer='uniform', activation=tf.nn.relu))\n",
    "model.add(Dense(num_classes, kernel_initializer='uniform', activation=tf.nn.softmax))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "# defining the parameters for RMSprop (I used the keras defaults here)\n",
    "rms = RMSprop(lr=0.001, rho=0.9, epsilon=None, decay=0.0)\n",
    "\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer=rms,\n",
    "              metrics=['acc'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4 - Deciding on the `batch size` and `number of epochs`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 60000 samples, validate on 10000 samples\n",
      "Epoch 1/60\n",
      "60000/60000 [==============================] - 1s 14us/step - loss: 1.1320 - acc: 0.7067 - val_loss: 0.5628 - val_acc: 0.8237\n",
      "Epoch 2/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.4831 - acc: 0.8570 - val_loss: 0.3674 - val_acc: 0.8934\n",
      "Epoch 3/60\n",
      "60000/60000 [==============================] - 1s 9us/step - loss: 0.3665 - acc: 0.8931 - val_loss: 0.3199 - val_acc: 0.9061\n",
      "Epoch 4/60\n",
      "60000/60000 [==============================] - 1s 9us/step - loss: 0.3100 - acc: 0.9092 - val_loss: 0.2664 - val_acc: 0.9233\n",
      "Epoch 5/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.2699 - acc: 0.9206 - val_loss: 0.2295 - val_acc: 0.9326\n",
      "Epoch 6/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.2391 - acc: 0.9305 - val_loss: 0.2104 - val_acc: 0.9362\n",
      "Epoch 7/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.2115 - acc: 0.9383 - val_loss: 0.1864 - val_acc: 0.9459\n",
      "Epoch 8/60\n",
      "60000/60000 [==============================] - 1s 9us/step - loss: 0.1900 - acc: 0.9451 - val_loss: 0.1658 - val_acc: 0.9493\n",
      "Epoch 9/60\n",
      "60000/60000 [==============================] - 1s 9us/step - loss: 0.1714 - acc: 0.9492 - val_loss: 0.1497 - val_acc: 0.9538\n",
      "Epoch 10/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.1565 - acc: 0.9539 - val_loss: 0.1404 - val_acc: 0.9591\n",
      "Epoch 11/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.1443 - acc: 0.9569 - val_loss: 0.1305 - val_acc: 0.9616\n",
      "Epoch 12/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.1334 - acc: 0.9596 - val_loss: 0.1224 - val_acc: 0.9628\n",
      "Epoch 13/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.1257 - acc: 0.9627 - val_loss: 0.1133 - val_acc: 0.9660\n",
      "Epoch 14/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.1169 - acc: 0.9652 - val_loss: 0.1116 - val_acc: 0.9674\n",
      "Epoch 15/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.1091 - acc: 0.9675 - val_loss: 0.1104 - val_acc: 0.9670\n",
      "Epoch 16/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.1051 - acc: 0.9689 - val_loss: 0.1030 - val_acc: 0.9692\n",
      "Epoch 17/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0978 - acc: 0.9697 - val_loss: 0.1044 - val_acc: 0.9686\n",
      "Epoch 18/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0929 - acc: 0.9718 - val_loss: 0.0996 - val_acc: 0.9689\n",
      "Epoch 19/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0882 - acc: 0.9738 - val_loss: 0.1035 - val_acc: 0.9695\n",
      "Epoch 20/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0850 - acc: 0.9737 - val_loss: 0.0941 - val_acc: 0.9717\n",
      "Epoch 21/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0803 - acc: 0.9751 - val_loss: 0.0953 - val_acc: 0.9715\n",
      "Epoch 22/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0793 - acc: 0.9762 - val_loss: 0.0898 - val_acc: 0.9729\n",
      "Epoch 23/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0747 - acc: 0.9775 - val_loss: 0.0901 - val_acc: 0.9732\n",
      "Epoch 24/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0718 - acc: 0.9778 - val_loss: 0.0948 - val_acc: 0.9720\n",
      "Epoch 25/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0697 - acc: 0.9781 - val_loss: 0.0908 - val_acc: 0.9727\n",
      "Epoch 26/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0668 - acc: 0.9794 - val_loss: 0.0917 - val_acc: 0.9726\n",
      "Epoch 27/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0648 - acc: 0.9800 - val_loss: 0.0895 - val_acc: 0.9737\n",
      "Epoch 28/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0637 - acc: 0.9798 - val_loss: 0.0868 - val_acc: 0.9728\n",
      "Epoch 29/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0598 - acc: 0.9813 - val_loss: 0.0883 - val_acc: 0.9736\n",
      "Epoch 30/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0570 - acc: 0.9820 - val_loss: 0.0869 - val_acc: 0.9741\n",
      "Epoch 31/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0555 - acc: 0.9825 - val_loss: 0.0896 - val_acc: 0.9732\n",
      "Epoch 32/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0554 - acc: 0.9827 - val_loss: 0.0843 - val_acc: 0.9743\n",
      "Epoch 33/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0512 - acc: 0.9836 - val_loss: 0.0843 - val_acc: 0.9746\n",
      "Epoch 34/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0509 - acc: 0.9835 - val_loss: 0.0868 - val_acc: 0.9753\n",
      "Epoch 35/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0491 - acc: 0.9842 - val_loss: 0.0841 - val_acc: 0.9755\n",
      "Epoch 36/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0471 - acc: 0.9848 - val_loss: 0.0887 - val_acc: 0.9728\n",
      "Epoch 37/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0466 - acc: 0.9850 - val_loss: 0.0876 - val_acc: 0.9756\n",
      "Epoch 38/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0456 - acc: 0.9856 - val_loss: 0.0833 - val_acc: 0.9769\n",
      "Epoch 39/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0431 - acc: 0.9866 - val_loss: 0.0869 - val_acc: 0.9759\n",
      "Epoch 40/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0413 - acc: 0.9869 - val_loss: 0.0926 - val_acc: 0.9743\n",
      "Epoch 41/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0401 - acc: 0.9872 - val_loss: 0.0851 - val_acc: 0.9756\n",
      "Epoch 42/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0401 - acc: 0.9876 - val_loss: 0.0856 - val_acc: 0.9764\n",
      "Epoch 43/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0392 - acc: 0.9870 - val_loss: 0.0861 - val_acc: 0.9771\n",
      "Epoch 44/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0396 - acc: 0.9870 - val_loss: 0.0918 - val_acc: 0.9756\n",
      "Epoch 45/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0371 - acc: 0.9883 - val_loss: 0.0866 - val_acc: 0.9766\n",
      "Epoch 46/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0374 - acc: 0.9883 - val_loss: 0.0888 - val_acc: 0.9748\n",
      "Epoch 47/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0376 - acc: 0.9878 - val_loss: 0.0850 - val_acc: 0.9761\n",
      "Epoch 48/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0351 - acc: 0.9890 - val_loss: 0.0848 - val_acc: 0.9777\n",
      "Epoch 49/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0361 - acc: 0.9884 - val_loss: 0.0850 - val_acc: 0.9771\n",
      "Epoch 50/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0341 - acc: 0.9887 - val_loss: 0.0889 - val_acc: 0.9769\n",
      "Epoch 51/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0322 - acc: 0.9897 - val_loss: 0.0882 - val_acc: 0.9771\n",
      "Epoch 52/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0322 - acc: 0.9895 - val_loss: 0.0892 - val_acc: 0.9762\n",
      "Epoch 53/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0313 - acc: 0.9892 - val_loss: 0.0916 - val_acc: 0.9771\n",
      "Epoch 54/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0300 - acc: 0.9897 - val_loss: 0.0913 - val_acc: 0.9772\n",
      "Epoch 55/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0306 - acc: 0.9902 - val_loss: 0.0904 - val_acc: 0.9763\n",
      "Epoch 56/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0307 - acc: 0.9900 - val_loss: 0.0910 - val_acc: 0.9777\n",
      "Epoch 57/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0299 - acc: 0.9901 - val_loss: 0.0918 - val_acc: 0.9763\n",
      "Epoch 58/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0299 - acc: 0.9906 - val_loss: 0.0914 - val_acc: 0.9778\n",
      "Epoch 59/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0284 - acc: 0.9909 - val_loss: 0.0907 - val_acc: 0.9769\n",
      "Epoch 60/60\n",
      "60000/60000 [==============================] - 0s 8us/step - loss: 0.0283 - acc: 0.9908 - val_loss: 0.0962 - val_acc: 0.9761\n",
      "CPU times: user 1min 17s, sys: 7.4 s, total: 1min 24s\n",
      "Wall time: 29.3 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "batch_size = input_dim\n",
    "epochs = 60\n",
    "\n",
    "model_history = model.fit(x_train, y_train,\n",
    "                    batch_size=batch_size,\n",
    "                    epochs=epochs,\n",
    "                    verbose=1,\n",
    "                    validation_data=(x_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test loss: 0.09620895183624088\n",
      "Test accuracy: 0.9761\n"
     ]
    }
   ],
   "source": [
    "score = model.evaluate(x_test, y_test, verbose=0)\n",
    "print('Test loss:', score[0])\n",
    "print('Test accuracy:', score[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAApcAAAGJCAYAAADfWBjnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3Xd81eX5//HXRQh7yhYEZIlVECpO3LS4Z11ttT8nxVGrnVStdde22lqtYrVaZ79qtWprlVoHWEBUqNbBEEHCUtmEQBiS6/fH/fmYk5NzkpzknOSEvJ+Px+dxn/OZ9zkEcnHdy9wdEREREZFsaNbQFRARERGRHYeCSxERERHJGgWXIiIiIpI1Ci5FREREJGsUXIqIiIhI1ii4FBEREZGsUXApIiIiIlmj4FJEREREskbBpYiIiIhkTYMHl2Z2qpndaWb/MbNiM3Mze7SW9+pjZg+Y2XIz22Jmi8zsdjPrXMU1XzGzJ81shZltNrN5ZnadmbWu4poDzewFM1tjZpvM7D0zu9zMCmpTbxEREZEdhTX08o9m9i6wF1ACLAWGAo+5+1kZ3mcgMB3oDjwHzAX2BQ4H5gGj3X110jX7Aa8ChcBTwBLgCGAUMA0Y4+5bkq45EXga2Aw8AawBjgd2A55y99NqWueuXbt6//79M/mYIiIiIg1i1qxZq9y9W3XnNa+PylTjCkJQ+TFwKPBaLe9zNyGwvMzd74x3mtlvo2fcBIxP2F8A/BloA5zo7n+P9jcDngS+EV13S8I1HYD7gO3AYe4+M9r/c0KQeqqZnenuj9ekwv3792fmzJm1/LgiIiIi9cfMimpyXoM3i7v7a+4+3+uQQjWzAcBYYBFwV9LhXwAbgbPNrG3C/kOB3YHX48Ayqk8Z8JPo7Xgzs4RrTgW6AY/HgWV0zWbg6ujtRbX9HCIiIiKNXYMHl1lyRFS+FAWHX3L3DYQm7jbA/imumZR8M3dfCHwE9AMG1OQa4HVgE3CgmbXM9AOIiIiI7Ah2lOByt6j8KM3x+VE5JFfXuPsXwCeErgYDko+LiIiINAX50OcyGzpG5fo0x+P9nRrgmoxs27aNpUuXsnnz5treoklr1aoVffr0obCwsKGrIiIi0iTtKMFldeJ+k5n068zJNWY2DhgH0Ldv30rHly5dSvv27enfvz8Vu3tKddyd1atXs3TpUnbdddeGro6IiEiTtKM0i8cZw45pjndIOq8+r6nA3e9191HuPqpbt8qj+Tdv3kyXLl0UWNaCmdGlSxdlfUVERBrQjhJczovKIWmOD47KxL6SWb3GzJoDuwJfAAurqmx1FFjWnr47ERGRhrWjBJfx3Jhjo3kqv2Rm7YHRQCkwI+HQq1F5VPLNoqmNhgBFVAwU014DHEIYkT49eeJ1ERERkaaiUQWXZlZoZkOj1Xi+5O4LgJeA/sAlSZddB7QFHnb3jQn7pwBzgEPM7ISEZzQDfhW9vSdp/s2ngFXAmWY2KuGaVsCN0duJtfx4eWHdunXcfffdGV93zDHHsG7duhzUSERERBqTBh/QY2YnASdFb3tG5QFm9mD0epW7/yh63ZsQEBYRAslEFxOWf7zDzMZE5+1HWP7xI+CqxJPdfbuZnUvIRj5lZk8Bi4ExlC//+Luka4rN7EJCkDnZzB4nLP94AtHyj4QlIRutOLi8+OKLK+zfvn07BQXpl05/4YUXcl01ERERaQQaPLgERgD/L2nfAMrniiwCfkQ13H1BlE28ntBsfQzwKXAHcJ27r0lxzZtmtg8huzkWaB8973rgllTN2+7+rJkdSghWvwG0Iixd+QPgjrqsNFTJ5ZfDu+9m7XYAjBgBt9+e9vCECRNYsGABI0aMoLCwkHbt2tGrVy/effddZs+ezUknncSSJUvYvHkz3//+9xk3bhxQvpRlSUkJRx99NAcddBDTp0+nd+/ePPfcc7Ru3Trl8+677z7uvfdetm7dyqBBg3jkkUdo06YNn3/+OePHj2fhwtArYeLEiRx44IE8/PDD3HrrrZgZw4cP55FHHsnu9yMiIiJ1YtmMhSQzo0aN8uS1xefMmcPuu+8e3jRAcLlo0SKOO+44PvjgAyZPnsyxxx7LBx988OXUPmvWrGGnnXaitLSUffbZhylTptClS5cKweWgQYOYOXMmI0aM4PTTT+eEE07grLPOSvm81atX06VLFwCuvvpqevTowfe+9z3OOOMMDjjgAC6//HK2b99OSUkJS5cu5ZRTTmHatGl07dr1y7okq/AdioiI5Dt32LoVNm2CjRsrl+7QogUUFpZvye979Aj7csjMZrn7qOrOy4fMpaRTRRBYX/bdd98Kc0becccdPPPMMwAsWbKE+fPnfxkcxnbddVdGjBgBwN57782iRYvS3v+DDz7g6quvZt26dZSUlHDkkUcC8Oqrr/Lwww8DUFBQQMeOHXn44Yc59dRT6dq1K0DKwFJERBopd1i7FjZsgJYtK25VdMvK2BdfhGcUF4cy3euNG6F584oBXHJQ16xZuN8XX8D27eWv4/fbtoX7pLp/4r5Nm8L5dTFjBuy3X3a+ozpScClVatu27ZevJ0+ezMsvv8wbb7xBmzZtOOyww1LOKdmyZfnS6gUFBZSWlqa9/znnnMOzzz7LXnvtxYMPPsjkyZPTnuvummpIRKS+uMPKlbBgQdhWrYLOnaFLl4pb584hyEp3j61bobQ0BFAbNsDSpbBkCSxeXF7GrzduTH2fgoIQZLZqVV4mbq1bV3zfrFnlQC7eNm2q2ecvLAwBX1lZ7b6/WIsW0L49dOgQyvbtoWtX2HXX8vdt20KbNulLsxCobt0aynhLfD8gf1aeVnApFbRv354NGzakPLZ+/Xo6d+5MmzZtmDt3LjNmzEh5XiY2bNhAr1692LZtG4899hi9e/cGYMyYMUycOPHLZvGNGzcyZswYTj75ZK644gq6dOmStllcRKTBbd8egpjWrUP2K1s2bYL582HevPLtk0/Ks17xf8CTyxYtQnDTsWMoE1937BiCmM8+Kw8k462kpPo6mZUHnQUFoY5xMFlaWnVw1rMn7LIL7LEHHH10eN2xI2zZUnnbvLni+9LSsC/eiovLX2/fXh7Qde8OgwaVv4+DvOrKeBnhOAOZKqgrKwvnNW8etoKC8tfx+ya4HLGCS6mgS5cujB49mj333JPWrVvTo0ePL48dddRR3HPPPQwfPpzddtuN/fffv87Pu+GGG9hvv/3o168fw4YN+zKw/f3vf8+4ceO4//77KSgoYOLEiRxwwAFcddVVHHrooRQUFDBy5EgefPDBOtdBRJqQuOl13boQnMQBSqqyun1xf7hU25aE8aDt2kGnTiEA69Sp4uv27UOWzay8TH796aflgWRRUcXP07cvDBwYgsd4DEVyCaHORUWwfn0IwtavT90M27JlyKgNHAiHHhrKeOvWLXxvq1dX3latCqV7yLS1bl2xjF+3bQt9+oQgsk+f8Lx8V1AQtlatGromjYYG9DSgagf0SK3oOxRporZtC8HPypWhiTXdVtNm0URx02vc/Bq/bts2/da6dQhC160LWxzUJr7esKH6Ztd27WC33SpuQ4fC4MEhaKsN91C3uKl4w4aQ4evdO30TtzR5GtAjIiL5yz1k05KDrg0b0jeDxvvWry+/Zu3a8i1Vfz0z6NUrZMqGDYNjjgmvu3RJHzAmv27Zsrx5OdffSVlZKBNf5+L5ZuUZxZ49qz9fJAMKLqVeXHLJJUybNq3Cvu9///uce+65DVQjEamRTZtCs+zq1eXNtQUFoUx+vXFj6ibTxC0xmNy6NbO6xCOHO3Ysb1YeMCC8jrdOnULguMsuYdt555xPz5I1ZtkdFS3SQBRcSr246667GroKIk2LexiMUVwcgr6tW0Pmb+vWyq83bw5NycuXh0By+fLybf36utWjXbswMrZLF9hpJ+jfv3K/w8T37dqVZwsTt8LC+skeikidKbgUEclX7iE4TDVwIt7WrAkBYOJAjbgfXaZ96gsLQ6Zv553hK1+Br30tNCnvvHMIEKF8apayssqv27SpOEXNTjs1jgEbIpJVCi5FRBqaexho8uGHYZs9u7xMNxWMWQjedtopZP06dAgrdKSaaqZt2xDktWgRtsTX8fs4u6jsoIjUkYJLEZFMuMOKFWEi6PXrQ9/BOHOY+Hr9+nBuPI1JPOdd4vvSUpgzp3IQ2bNnyByeey706xeCvjj4i7dOndQ/T0TykoJLEZFUNm+uPFl1vFXVD7F9+/KMYUFB+TJw8dJwiWVhYZhS5txzQzC5xx6hTFpSVUSkMVFwKXXSrl07SmqygoNIrm3YAB9/HJqXE1fuSF7FoyYTZMdL1CX2WezTJ8wv+K1vhTIemNKxY3kZB5QiO4gtW2DuXHjvvfBXKx6ov9NOFQfpd+6c3YWIci3+q51pLxD3MOZt7tywbdsW/jnYffcwOUFNpgjdvj0sqhT3glmxIjRWxN2d427OnTo13l4qjehHQUSavDiAnD+/cvn559Vf36JF1fMZdupUPjH2rruWT1g9eHAYxSx5p7Q0rFI4fz589FEoV64sn9oycevdO/zx5trGjeHHccWKUJdVq8rLxG3lyvLgZI89YM89y5PX7dunv39ZWbh/PCf86tUhAT5yZNXXVcU9TA7w3nsVt7lzQ6K9JuIlslu0CEn5xG698da8ecUVFBO3eF+LFiG46t27POBK3Hr0COPVEic1SN5WrSpvMEjc4rFnEOqa6v7x1rFjCALnzg29V+KAsrg49edv3bo80Bw6NGyDBoVg9MMP4YMPQjlnTvi5jbVrl7prdatW5XXp2TN87nRbbefSzxWt0NOAqluh5/LL4d13s/vMESPg9tvTH//pT39Kv379uPjiiwG49tprMTNef/111q5dy7Zt27jxxhs58cQTgaozlyUlJZx44okpr3v44Ye59dZbMTOGDx/OI488wueff8748eNZuHAhABMnTuTAAw/M+DNqhZ5GrKQEFi2qvH3ySSjXrKl4fq9e4V/vwYPDNmhQ6KMYr47SqlX51rKlVh7JAffQS2DVqvIZj5KDhXgzg4MPDr8sa6OoCP75z/AL+qOPwrZkScUEc48eYaGZTz8NdUrWtWsINIcMgeHDy7dddqk+S+QeAsI4iC0qCkFevH32WShTzeUO4UewW7dQh3hr1iwELLNnh6R5rG/f8mDTLHzOpUtDuWxZ+G6TmYWAZtSo8m3EiIqBx4YNoe6JwXj8Xa5dW/H5id/P8OHhr9aGDeG8NWsqzl8fvy8tTR80xq8TA89Ur0tLK86IVZPZsOKANN66dq3czTmekjUu16+vHJQm/hkk6t27PGBMDB4LC0NPmcTgc86c8LORHF717h3+PFP9RyKeTjZVoLxsWfnPWOKfUaJ27eCVV2Dffav/ruqipiv0KLhsQPkYXL7zzjtcfvnlTJkyBYCvfOUrTJo0iU6dOtGhQwdWrVrF/vvvz/z58zGzKoPLL774gk2bNlW6bvbs2ZxyyilMmzaNrl27smbNGnbaaSfOOOMMDjjgAC6//HK2b99OSUkJHTt2zPgzKrjMc+7ht2RyOmDOnPDbOVGrVqH5OXEbODAEkgMHKptI+Drr0nRWXBx+ERYXV91joKQkZMhSZeBqmtmKjRoFJ5wQtuHD09ffPQSSzzwDzz4L//1v2N+pUwgOhwwJPwpxOXhw6JkQKy0tD8jibelSWLw4/Mh98kn5uZ06VQykBg8Ov9STg7DErJVZ6B7bo0f6zFIcTHbrFoK8dJ81uak0znLNnRuOJy7HnZiN7dMnNEnPng0zZ4bt7bfL/yo1axaCmE6dwmdI/iu2yy7l392ee8Jee4WFjDp1qvmfZ65t3Fgx8Prss5BVTAwmd9qp7k3I8X+U4uesWVPegJH4c1UTmzaVN6zE4/M6d65b/SB0U1ixouJ/auLtxz8OAWwuKbhsBPJ1bfHdd9+dV155hZUrV3LxxRczefJkrrjiCl5//XWaNWvGvHnz+OSTT+jZs2eVweW2bdtSXvfXv/6Vzz77jJtuuqnC+d26dWPp0qW0rOO8ePnwHTYJ27eH34BvvBG2VatSr9oSv96+PfxLO29exTagjh3LUwFDhoR/zeNAskePan9jlJVV7EoJ4R/z+uz6GDdTLloUArVFi0IA06ZN5aa9Xr1SN2GVlqb+hbFmTcXpK5PLTZtC8JKuaa9Xr/BdFBWV1y2xnukyIcnimY+SM2/x+y5dwi/gVM2hiRmpf/0L/v53ePPN8Mu8b9/yQPPQQ0O26c03Q0D5zDPhRwbggAPgpJPg5JNDgjobfdGKi+H99ys3BSf+eJqFH8XEIDYu+/YNny2Xtm8vXxgpE8uXlwebM2eGjGPyZxg4MP+aUyW/aW1xqbVTTz2Vp556is8++4wzzzyTxx57jJUrVzJr1iwKCwvp378/m9O1HSRId527Y421l3JTtm5d+K0/fXrY3nwz/MaCEATuskvlybUTJ9mGEDied17F9qU0AWRpKcx/v2KCc968EAwlZtW2bKlc1RYtQkAwYED4BRpvAwaErbCwfD7yxExc/DoOuNI1qRUUhGxKHKAVFVWuR+fO5XVM1qlTeZ+ulStDEBl/lcnatas4ZWWHDuGrjve1bh3qHGdb/vvfcL90eYO2bcN3069fCNj69w9BUtzdNLEranKX1Gz0KvjqV+FnPwvZp3/+MwSa998Pf/hD+Ext2oRjzZvDEUfAD38IJ54YguRs69ABRo8OW6ysLPx5fvxxyAoOGNCw88DX9j9JO+9cHrSL1DcFl1LJmWeeyYUXXsiqVauYMmUKTz75JN27d6ewsJDXXnuNoqKiGt1n/fr1Ka8bM2YMJ598MldccQVdunT5sll8zJgxTJw48ctm8Y0bN9Ih07YIyQ730EY3dSpMmxa22bPD/mbNQrvhWWfBgQeGbdddUwaI27eHTNCUKfDWW+HywrXQYha0eL9yn6sNG8oDyU8+qTiis3//8oEP6cbjtG4dgoNFi8IgjwULQhycrgN+KvFoWLPKgwASt9atQ5322isEP3GytV+/sLVrF+q/bl36QQfr18Pee6dvUu3evXaBzRdfhAAzfs62beX1y5d50nv2hPPPD9umTaG/2N//Hn4Gjj8ejj22YZpmmzULP8677lr/zxbZUahZvAHla7M4wLBhw+jatSuvvfYaq1at4vjjj2fbtm2MGDGCadOm8eKLL9K/f/8qm8Wruu6hhx7iN7/5DQUFBYwcOZIHH3yQzz//nHHjxrFw4UIKCgqYOHEiBxxwQMZ1z5fvsFHZti108J02rTygjDtndewI++8f0jsHHhh6jKcZkrp9O/zvfyGYnDwZXn89BFcQsm2tW1ce5BG//+KLECDutlvljvNDhtR+lK97aFpesAAWLgzl9u2pm3d32ikEuSIiUpn6XDYC+RxcNmb6DsstWwaLi5zOzdbRefNndN60jBarlpf3jv/003DSu++G9BGE9FbcVnjQQaEnekEBpaWVR4bGr9esCUHl66+Xj+wcPDj0oTvssFD26VN1XeOWcw3oFhHJT+pzKdLUbNrE0ikLmPz3Yia/0ZIpH+/Mxxt3BgzoHG2705YSOrOWnZqto3PLTXRusxXv2ZHSdt0pbd2ZzWUtKf0fbH4TSm8K/QY3bUo/RQeEZtbBg+GMM0IgeeihmY9aVFApIrJjUHApdfb+++9z9tlnV9jXsmVL3nzzzQaqUeOVOEvP3Llh1Gq7dmFr2zbhtZfQbu5Mms18izffbsbkBbswuWRvFjAMgE6s5dB2s7h42CR2GwLr2/RiTfPurG3WhbXekTVbe7J2wy6sWQMfrw2BXdx3sWuKvoxt2lRelSPxdceOCg5FRCRQcJmHGtto6mHDhvFutifkrKXG0s3DPYx8nj27fDT0nDmVZ+lJrx1wWLRB5xYlHDp0KZfu/y6HHdeeYcfsQkHrr+Ws/iIiIukouMwzrVq1YvXq1XTp0qVRBZj5wN1ZvXo1rVq1auiqpPXxx/DII/Doo2FwSWyXXcLglfPOiwaxDN7O7lvepcNbL7Px5TcoeXsOG7c2p6RZR0qG7s3GPfajZPBItvYfwsh9Cxk2rB3Nmg1tuA8mIiISUXCZZ/r06cPSpUtZuXJlQ1elUWrVqhV9qhs5Us9Wr4YnnghB5YwZoX/imDFhrr+RI8Po6HbtCKOz//UvmDQJrnrpy6UOW48YQddLjwuT/h18cOZLRYiIiNQjBZd5prCwkF01wVqjt2ULPP98CChfeCFMtzNsGPz61/Ctb0WDXbZtC5Mw3jQpBJRx14IePcJEf0cdBV/7WpgnR0REpJFQcClSS6WlYaLvxPkT4+2TT8Lcjb16wWWXwdlnh8m22bYNXn4ZJvwFnnsuzBjdvHmY9ueXvwwB5fDhGh0jIiKNloJLkRpasgRefDEkGd96K0wPmah9+7DE4J57hhVbxowJW4GVhbW3L/kLPPlkWKuvUyc47TQ4Lmru7tixYT6UiIhIlim4FElj69awWM2LL4btww/D/j594PDDw7yOietWd+2atKzeBx/Az/8Cf/lLWKy4Vauw0O+3vhUylA25YLGIiEiOKLgUSbBlCzz8cOgv+corsHFjWPv6kEPgnHPg6KPDgjUpB/KXlcHbM8MCyc8+G6LRggL4+tfhhhvgpJPSLpsoIiKyo1BwKRJ56SW49FKYPz+sgPid74QE4xFHRKO5UykthVdfDf0n//GPMOK7oCAsm3jnnXD66dC9e31+DBERkQal4FKavCVL4Ior4OmnQ1P3pEkwdmya7CSEPpP/+EfIUL70UlgbsX37EImecEJIb3bpUq+fQUREJF/kzZBUM+tjZg+Y2XIz22Jmi8zsdjPrnOF9TjazV81snZltNrM5ZnaNmVWaWdvMrjUzr2ZbkHTNYdWcf0tdvwupH1u3wq9+BUOHhumCbrwR3n8fjjwyRWC5di38+c8hgOzZM8x2PmsWnHtumJty5cowWOessxRYiohIk5YXmUszGwhMB7oDzwFzgX2B7wNHmdlod19dg/vcAFwNlABPA6uBg4DrgLFm9nV3L024ZHIVtzse+CrwYprjU9JcP7W6ekrDe+01uOSSsOTiiSfC7beHpvAKiotDdvKJJ0IAuW0b7Lor/PjHYaT3yJFVpDdFRESaprwILoG7CYHlZe5+Z7zTzH4LXAHcBIyv6gZmNhK4ClgH7O3uC6P9BtwBXAr8FLg2vsbdJ5MiQDSzAuD86O29aR452d2vTXNM8tTy5fCjH8H//V+IE59/Ho49NuGEL76AZ54JJ7zwQhjh06cPfO97cOaZMGqUAkoREZEqNHizuJkNAMYCi4C7kg7/AtgInG1mbau51cmAAX+KA0sAd3fgSsCBi6LAsTrHAH2AGe7+Xk0+h+S3VavgJz+BQYPgb3+DX/wiDOb+MrB0D4Ny9torDMJ54w0YNy7MRVRUBLfdBvvso8BSRESkGvmQuTwiKl9y97LEA+6+wcymEYLP/YFXqrhPz6hcmHwgus8qQnZ0GPBuNXUaF5XpspYAg8zsUqAD8BnwH3efX819pZ6tXRviwt//Poy7+fa34dprYcCAhJOmTQuR5/TpMGQI/PWvcPLJYdS3iIiIZCQfgsvdovKjNMfnE4LLIVQdXK6KykoLc5tZeyBeoHkoVQSXZtYbOBpYDzxRxfO+HW2J1z4NXOjua6u4TupBcXEIKG+7DdavhzPOCEHl0KEJJ82eDT/7WehX2asX/PGPYaBO83z4ayEiItI4NXizOBCve7c+zfF4f6dq7vN8VF5gZv2Tjt1IaDIHqG70+QVAAfCou29KcXwlMIGQAW0PdCMEo+8A3wD+YWb58L02SZs2wW9+EzKT11wDhx0G//sfPP54QmC5ZAmcfz4MGwaTJ8NNN4XJLceNU2ApIiJSR43hN2kcFHpVJ7n7dDP7I/Bd4L0oi7gGGA3sA3wI7AFsT/ugEBSeF71N2STu7h9G94qVAJPMbDohIzqaMNL8uTTPGEfU7N63b9+qPpJkoKgozBR0zz3w+edhxqDrrw/dJL+0fHlIZd59d1hN5/LL4corNXWQiIhIFuVDhi3OTHZMc7xD0nlpuft4wijv2cDphBHmW4Ejgfej01ZUcYujgb7UYiCPuxcDf4neHlLFefe6+yh3H9WtW7dMHiFJtmwJU0seeWQY+X399WF2oHg98C8Dy4ULYfz4cNLtt4cBOx99FAJNBZYiIiJZlQ+Zy3lROSTN8cFRma5PZgXu/gDwQPJ+M/tT9PLtKi6PB/L8sSbPSmFlVFY3sl3q4MMP4f77wxrgq1dD375h9Pc550C/fgknvv8+3HJLaBNv3jz0p/zxj5NG84iIiEg25UNw+VpUjjWzZokjxqOBOKOBUmBGbR9gZmOBfsAUd1+W5pydgWMJGdIna/mo/aOy0oh1qbsnngiJxxkzoLAQTjopdJ382teSBnbPmAE33xyWaGzXDn7wg7C+4847N1jdRUREmooGbxZ39wXAS0B/4JKkw9cRsoAPu/vGeKeZDTWzoUnnYmYdUuwbSOg/uZ0wECed8wkDeR5JM5Anvt/oVAN2zOws4AxCM3xtg1NJ4557whzm69fDb38Ly5aVN4l/GVh+8AEccQQccECYXui660JnzN/8RoGliIhIPcmHzCXAxYTlH+8wszHAHGA/4HBCc/hVSefPicrkGa3vN7N+wCxgLTCIMLimELjA3VNmP6NgsboVeWKPAc2iATxLgVaEAUP7Al8A33X3RdXcQzLwzDNhqcbjjguvUw7ofuyxMNq7bdvQl3LcuJC1FBERkXqVF8Gluy8ws1HA9cBRhBVyPiUs23idu6+p4a2eJ/SbPJ0wTdAKwhrjv65mgM6RhGbzGe7+fhXnAUwEvkZoru9KCHCXAQ8Ct7v7/2pYV6mBqVPhm9+EffcNzeKVAsstW0Kz9913w8EHh5N69WqQuoqIiAhYWB1RGsKoUaN85syZDV2NvPXhh3DQQdCjRwgyu3ZNOmHxYjjtNHjrrbBg+M03h86YIiIiknVmNsvdR1V3Xl5kLkWSLVkS5qps3RomTUoRWP7rX2Etx61b4emn4ZRTGqSeIiIiUlGDD+gRSbZ2LRx9dFjC8cUXoX//hINlZWFCy6OPDs3fM2cqsBQREckjylxKXikthRNPDKsxTpr//hJmAAAgAElEQVQEe+2VcHD1ajj77BBxnnVWGELeVlOKioiI5BMFl5I3tm8PLd1Tp4Z5zw8/POHgvHkhW7lsWRi8M348WPJkASIiItLQFFxKXnCH730vTDV0xx1hhcYvvfUWHHMMNGsGr78O++3XYPUUERGRqqnPpTQo9zDf+emnw8SJMGFCCDK/9K9/hYnRO3QIJyqwFBERyWsKLqVBlJTAvffCyJFhuqGXXoKrrw6zCX3p0UfDzOmDB8P06aEUERGRvKbgUurVnDlw2WXQuzd897th3733wvLlcMMNCd0ob7stDN45+GCYMgV69mywOouIiEjNqc+l5Jw7PPss3HknvPYatGgR5j6/+OKwDHiFcTllZfDTn8Ktt4aTHnkEWrZssLqLiIhIZhRcSs7deCNccw306we//CWcdx50757ixG3bwsFHHw2Lif/+91BQUO/1FRERkdpTcCk59Yc/hMDyO9+BBx6oIlYsKQmZykmTQjR65ZWaakhERKQRUnApOfPYY2Hk9wknwP33VxFYbt0aZk6fPBnuuw8uuKA+qykiIiJZpOBScuIf/4D/9//CROhPPAHN0/2kucP558Orr8JDD4UUp4iIiDRaGi0uWTdlSpi3cuRIeO45aNWqipN//vPQx/KGGxRYioiI7AAUXEpW/fe/cPzxsOuuYQnw9u2rOPm+++Cmm0Iz+FVX1VsdRUREJHcUXErWzJ0LRx4JO+0UJkXv2rWKk198ES66CI46KqwVrsE7IiIiOwQFl5IVixfD2LFh+e9//xv69Kni5P/+N4wMHz4cnnwSCgvrrZ4iIiKSWxrQI3W2YgV8/etQXBwGfFe5SuOiRXDssdClCzz/fDXt5iIiItLYKLiUOnGHb34TliwJTeEjRlRx8tq1cMwxUFoKL78MO+9cb/UUERGR+qHgUupk8uQwi9Add8BBB1Vx4pYtcPLJ8PHHIQrdY4/6qqKIiIjUIwWXUmvucO21IQF54YVVnFhWBuecE+YoeuwxOOyw+qmgiIiI1DsFl1JrkyfD66/DnXdWM5flb38Ljz8eph361rfqq3oiIiLSADRaXGolMWtZ5WqN06fDhAlwyinws5/VV/VERESkgShzKbVSo6zl6tVwxhnQt29YXFxzWYqIiOzwFFxKrVSbtSwrC4uLr1gRspedOtVn9URERKSBKLiUjMVZyzvuqCJredtt8M9/htTm3nvXZ/VERESkAanPpWTs2muhV68qRohPnx76V556KlxySX1WTURERBqYMpeSkcmTw4xCabOWq1aFfpb9+sGf/qR+liIiIk2MgkvJSJVZy8R+lm+8AR071nf1REREpIEpuJQaqzZreeut8MILcNdd8NWv1nf1REREJA+oz6XU2HXXVZG1nDYNrrwSTjsNLrqo3usmIiIi+UGZS6mRyZPD9vvfp8harloFZ54J/fvDffepn6WIiEgTpuBSaqTKrOV3vxv6Wc6YoX6WIiIiTVzeNIubWR8ze8DMlpvZFjNbZGa3m1nnDO9zspm9ambrzGyzmc0xs2vMLOWMjGbmVWwzqnjOcWY22czWm1mJmb1pZv8v08/dGMRZywkToHXrpIPTpsHf/gbXXAMjRzZA7URERCSfmLs3dB0ws4HAdKA78BwwF9gXOByYB4x299U1uM8NwNVACfA0sBo4KLrXNODr7l6adI0DRcCDKW651N3/lOI5lwJ3Rvd/AtgKnAr0AW5z9x9V+6GBUaNG+cyZM2tyaoM6/HCYNw8WLEgKLt3h0ENh/nz4+GNo27bB6igiIiK5ZWaz3H1UdeflS7P43YTA8jJ3vzPeaWa/Ba4AbgLGV3UDMxsJXAWsA/Z294XRfgPuAC4Ffgpcm+LyRe6ean+q5/QHbgXWAKPcfVG0/3rgbeCHZva0u79Rk/vlu1deKe9rWSlrOWkS/Oc/cPfdCixFREQEyINmcTMbAIwFFgF3JR3+BbARONvMqoteTgYM+FMcWAJ4SM1eCThwkZkV1LHK5wEtgT/EgWX0nLXAzdHbKgPhxqKkBMaNgwEDUvS1LCsLq/AMGADnn98g9RMREZH8kw+ZyyOi8iV3L0s84O4bzGwaIfjcH3ilivv0jMqFyQei+6wiZEeHAe8mndLJzM6L7rEemOXu6fpbxvWdlOLYi0nnNGoTJsAnn4S5LStlLZ94Av73P3jsMWjRokHqJyIiIvmnwTOXwG5R+VGa4/Ojckg191kVlbsmHzCz9kDX6O3QFNfuBdxPaH7/A/CGmb1rZsMyqa+7f0rItPYxszbV1DevvfpqmAv9ssvg4IOTDm7bBj//OQwfHqYgEhEREYnkQ3AZz12zPs3xeH+nau7zfFReEPWLTHQjockcIHn0+W+B0UA3oD2wD/AUIeB81cx617K+jXZOng0bQkv3oEFw880pTrj//jC65+aboVk+/AiJiIhIvsiHZvHqxEFhlcPa3X26mf0R+C7wnpk9TRh0M5oQMH4I7AFsT7ruh0m3mgmcZmZPAd8AfkQYVJSV+prZOGAcQN++fTO4bf35yU+gqCiM1WmTnH/dtAmuvx4OOgiOOaZB6iciIiL5Kx/STtVl+joknZeWu48HzgdmA6cTBtZsBY4E3o9OW1HDet0TlYfUsr7Faep4r7uPcvdR3bp1q2FV6s/LL8M998AVV8Do0SlOuPNO+PRT+OUvtRKPiIiIVJIPmct5UZmuT+XgqEzXJ7MCd38AeCB5v5nF81W+XcN6rYzK5FHq8wj9N4cAFaYbMrNe0flL3X1TDZ+TN4qLQ3P4kCFw440pTli7Fm65BY49NmQuRURERJLkQ+bytagca2YV6hMNxBkNlAJpV8upjpmNBfoBU9x9WQ0v2z8qk0efvxqVR6W45uikcxqVH/0Ili6Fhx5KMToc4De/gfXr03TEFBEREcmD4NLdFwAvAf2BS5IOX0fIBD7s7hvjnWY21Mwqjfo2sw4p9g0E7iX0tZyQdOyrqebPNLPhhJHjAI8mHf4zsAW4NHHgULRM5ZXR23toZP71L7jvPvjhD2H//VOc8OmncPvt8M1vhlHiIiIiIink6/KPc4D9CMs/fgQcmLj8Y7RkI+5uSff5KyFDOQtYCwwCjgcKgQvc/cGk8x8ETiFkGpcQgsahhKxkAXAf8F1P+pLM7HuEVX92iOUf16+HPfeEdu3gnXegVapV2C++OESfc+fCwIH1XkcRERFpWI1q+Ud3X2Bmo4DrCYHdMcCnhADuOndfU8NbPU8YiX06YVqhFYQ1xn/t7u+lOP9ZwgCc4YSJz1sRAsYXgfvc/e9p6nunmS0ijCT/DiEDPBu42t0fqmFd88YPfgDLl8Mbb6QJLBcsCIHlhRcqsBQREZEq5UXmsqnKh8zlCy+E8TkTJoQB4Cl9+9vwzDMhyOzVq17rJyIiIvmhppnLBu9zKQ3rkktgjz3g2mvTnPD++/CXv8DllyuwFBERkWopuGzC1q2DRYvgnHOgZcs0Jz3zTJjP8gc/qMeaiYiISGOl4LIJW7w4lP36VXHS1KkwbBh07VrFSSIiIiKBgssmrKgolGlXofziizDKJ+VSPSIiIiKVKbhswqrNXL7/PpSUaDUeERERqTEFl01YURG0aAHdu6c5YerUUCpzKSIiIjWk4LIJW7w4NIk3S/dTMG0a9OlTRbu5iIiISEUKLpuwoqIq4kb3kLkcPTqMFhcRERGpAQWXTdjixVX0t1y8GJYtU39LERERyUiNg0szG5nLikj92roVPv20isyl+luKiIhILWSSuZxlZm+a2Xlm1iZnNZJ6sXRpaPlOm7mcNg3atw9zXIqIiIjUUCbB5QvAV4H7gOVmdqeZKfJopOI5LtMGl1OnwgEHQPPm9VYnERERafxqHFy6+3FAf+AGoBi4BHjXzKaZ2XfMrFVuqii5UOUE6uvWwQcfqElcREREMpbRgB53X+bu1xKCzBOBF4F9gT8Dy8zsd2a2e7YrKdkXT6C+yy4pDr7xRmgz12AeERERyVCtRou7e5m7/yMhm3k9sBW4DPjAzCab2anZq6ZkW1ER9OwJLVumODhtGhQUwH771Xu9REREpHHLxlREewDDgS6AAauBg4EnzGyWmfXPwjMky6qchmjqVBg5Etq2rdc6iYiISONXq+DSzLqb2QQzW0BoGj8JmAycAvQEBgF/BEYAd2enqpJNaSdQ37oV3npL/S1FRESkVjIaCmxmY4DvEvpbFgJrgduBie7+ccKpnwAXm1lL4PQs1VWyxD1kLk88McXBd96B0lL1txQREZFaqXFwaWbzgQGEpu+ZhIzk4+6+uYrL5gNqW80zK1bAli1pMpfTpoVSmUsRERGphUwyl72BB4G73X1WDa95DHgj00pJbsUjxVP2uZw6FQYMgF696rVOIiIismPIJLjc2d3XZXJzd18CLMmsSpJraSdQdw+ZyyOPrPc6iYiIyI4hk0nUMwosJX/FmctKzeIffxzazNXfUkRERGqpxsGlmY03swVmtnOa472j4+dnr3qSC0VFYdnwTp2SDqi/pYiIiNRRJlMRfQv41N2Xpzro7suApcBZ2aiY5E48DZFZ0oGpU6FzZ9hdiyyJiIhI7WQSXO4G/K+ac94Dhta+OlIf0k6gPnUqHHggNMvG3PoiIiLSFGUSRXQEqut3WQx0rn11pD6knEB95UqYN0/9LUVERKROMgkuPyUs81iV4cDK2ldHcq2kBNasSZG5nD49lAouRUREpA4yCS5fA44ys5TRh5kdDBwNvJKNiklupB0pPm0atGgBo0bVe51ERERkx5FJcPkrYCvwspn91szGmtkeUfk74N/Alug8yVNpJ1CfOjUElq1a1XudREREZMeRyTyX8wjrhG8BLgdeJAzgeRH4PrAZOM3d5+SgnpIlKSdQLy2FmTM1BZGIiIjUWSYr9ODu/zSzAcA5wH5AJ8IgnxnAQ+6+Ous1lKxavBiaN09a3XHmTNi2Tf0tRUREpM4yCi4BogDythzURepBURH06QMFBQk748nTDzywQeokIiIiOw5NaNjEpJyGaOpUGDoUunZtkDqJiIjIjqNWwaWZ9TGz/czskFRbHe75gJktN7MtZrbIzG43s4zmzTSzk83sVTNbZ2abzWyOmV1jZpVGqkRLVn7PzF6MnrfFzFab2b/N7JQ09z/MzLyK7ZbafP76UmkC9bKyMA2R+luKiIhIFmTULG5mY4HfUf0qPAXVHE++70BgOtAdeA6YC+xLGCh0lJmNrkl/TjO7AbgaKAGeBlYDBwHXAWPN7OvuXppwyfeAnwKfEKZa+gzoB5wCfM3MfufuP0jzuCnA5BT7p1ZXz4byxRewbFlS5nLOHFi7Vv0tRUREJCtqHFya2X7A84RJ0v9ACMymAPOAg4Hdgb8D79SiHncTAsvL3P3OhGf+FrgCuAkYX039RgJXEQYY7e3uC6P9BtwBXEoIJK9NuOwt4DB3n5J0r90Jg5SuMLPH3H1WikdOdvdrU+zPW8uXw/btSZnLuL+lMpciIiKSBZk0i19JmG5oH3f/frTvNXcfD+wJ3AB8DXgqkwpEo8/HAouAu5IO/wLYCJxtZm2rudXJgAF/igNLAHf3qO4OXGRmBQnH/pYcWEb75wBPRG8Py+Tz5LN4GqIKmcupU6F7dxg0qEHqJCIiIjuWTILLA4C/u/vy5Os9+AUwh9AEnYkjovIldy9LPODuG4BpQBtg/2ru0zMqFyYfiO6zipAdHVbDem2Lyi/SHB9kZpea2ZVmdp6ZDa7hfRtMygnUp00LWUuzBqmTiIiI7FgyCS47AosT3m8FkrOJ04BMB/TsFpUfpTk+PyqHVHOfVVG5a/IBM2sPxEOhq+svipl1AL5ByHa+lOa0bwN3Eprs7wc+MrOnMh2AVJ8qZS7dYdEi+MpXGqpKIiIisoPJJLhcAXROej8w6ZxCoHWGdegYlevTHI/3d6rmPs9H5QVm1j/p2I2EJnOo+Bkqifpo/gnoAUxMseLQSmACIQPaHuhGWFP9HUJA+g8zS/u9mtk4M5tpZjNXrlxZzUfKrsWLw2xDbdpEOzZtCqPFO3as8joRERGRmsokuPyIisHkDODrZjYEwMx6EoKr+SmurYs4KPSqTnL36cAfCcHje2b2ZzO7zcxmEAbzfBidur2a590GnAb8B6g0UtzdP3T3X7n7B+5e4u6r3H0SoW/mJ8Bo4Pgq6nmvu49y91HdunWrpirZVVSU1CReXBzK9u3rtR4iIiKy48okuJwEHGpmO0Xvf0/IUr5jZm8Tpg/qBtyeYR3izGS69FmHpPPSigYXnQ/MJqyDPp7QfH8k8H502op015vZbwij018HjnH3LdU9M+HZxcBfore1musz1xYvThrMs2FDKDt0SHm+iIiISKYyCS7/SAiatgG4+zRChu8TwmjxT4GL3P3hDOswLyrT9amMB8qk65NZgbs/4O77u3vbaDvE3V8mDEgCeDvVdWb2O+BHhPkuj3b3kppVv4K4nbu6ke31zl2ZSxEREcm9Gs9zGWXm3kza9wzwTB3r8FpUjjWzZokjxqOBOKOBUkIzfK1Ek7/3A6a4+7KkY0aYt/Ni4N/AiUkTrWciHtFeacR6Q1uzBjZuVOZSREREcqvGmctoacYrsl0Bd19AGJHdH7gk6fB1hCzgw+6+MaEuQ82s0qjvaJR38r6BwL2EvpYTko5ZdOxi4EXghOoCSzMbnWrAjpmdBZxBaIZ/sqp7NISU0xApcykiIiJZlsnyj98iLP2YCxcTln+8w8zGEObL3A84nNAcflXS+fEI7uTJGe83s37ALGAtMIgwuKYQuMDdk7Of1wAXEDKj7wITrPJ8j++6+7MJ7x8DmpnZdGAp0ArYh7Bc5RfAd919Uc0+dv1JOYG6MpciIiKSZZkEl4sIk5BnnbsvMLNRwPXAUcAxhD6cdwDXufuaGt7qeWAcYTBPe8LgnaeBX7v7eynOj+fEbA38LM09HwISg8uJhJWIRhPmzjRgGfAgcLu7/6+Gda1XylyKiIhIfcgkuPwLMN7MOrv72mxXxN2XAOfW8NyUy8m4+0OEYLCmzzwHOKem50fX/Ar4VSbX5IOiImjdOsxz+SVlLkVERCTLMhkt/ktgJvCamR1nZj1yVCfJgXgaogqt/sXFUFAArVo1WL1ERERkx5JJ5nJzVBrwHECK/okQlhrP5L5SDypNQwQhc9mhg9YVFxERkazJJAj8D9WskiP5a/Fi2GuvpJ3FxepvKSIiIlmVyTyXh+WwHpJDmzfD559XkbkUERERyZJM+lxKI7VkSSgrTEMEylyKiIhI1im4bALiOS6VuRQREZFcq3GzuJldU8NT3d1vqGV9JAdSTqAOIXO5yy71Xh8RERHZcWUyoOfaKo7FA30seq3gMo8sXgzNmkGfPkkHlLkUERGRLMskuDw8zf5OhOUPLwP+CdxT10pJdhUVwc47Q2Fh0gH1uRQREZEsy2S0+JQqDj9nZk8AbwGP17lWklXxBOoVuCtzKSIiIlmXtQE97v4+YXL1K7N1T8mOlBOob9wYAkxlLkVERCSLsj1afDGwZ5bvKXVQVhamIqqUudS64iIiIpID2Q4u9wNKs3xPqYPPPoNt21JkLouLQ6nMpYiIiGRRJlMRJee+Eu+xC3AhcBDwZBbqJVmyeHEolbkUERGR+pDJaPFFVL22uAHzgR/VpUKSXWknUFfmUkRERHIgk+DyYVIHl2XAWsJI8efcfUs2KibZocyliIiI1KdMpiI6J4f1kBwpKoJOnVLEkMpcioiISA5obfEdXMppiECZSxEREcmJGgeXZjbQzL5jZl3SHO8aHR+QvepJXaWcQB2UuRQREZGcyCRzOQG4DShOc3w9cCvw47pWSrKnysxl8+bQqlW910lERER2XJkEl4cBL7v7tlQHo/3/Bo7IQr0kC9avD1vazGX79mBW7/USERGRHVcmwWVvwnREVVkM7Fzr2khWxSPFU2Yui4vV31JERESyLpPgcitQXTTSnqrnwpR6lHYaIgjN4upvKSIiIlmWSXD5AXCsmRWmOmhmLYDjgNnZqJjUXdoJ1EGZSxEREcmJTILLR4G+wJNm1jPxQPT+ScIykA9nr3pSF4sXQ4sW0KNHioPKXIqIiEgOZLJCz73AN4ATga+b2XvAMkJfzOFAG+Bl4J5sV1Jq54ADoKAAmqX6L0RxcZqUpoiIiEjtZbJCT5mZHQNcB1wE7J9weB1wO3Cdu5dlt4pSWyeeGLaUlLkUERGRHMgkcxlPN3SlmV0NDAU6EQLLuQoqGxn1uRQREZEcyCi4jEWBpAbuNFZlZVBSosyliIiIZJ2Wf2yKNm4Ed2UuRUREJOu0/GNTtGFDKJW5FBERkSzT8o9NUXH0/wNlLkVERCTLtPxjU6TMpYiIiORI3iz/aGZ9zOwBM1tuZlvMbJGZ3W5mnTO8z8lm9qqZrTOzzWY2x8yuMbNWVVzzFTN70sxWRNfMM7PrzKx1FdccaGYvmNkaM9tkZu+Z2eVmVpBJfRuEMpciIiKSI3mx/KOZDQRmAecCbwG/AxYC3wfeSDeIKMV9bgD+BuwDPAvcRegjeh3wcqpg0cz2A94GTiJMAv/76JprgH+bWcsU15wIvA4cAjwTPadFVO/Ha/q5G4wylyIiIpIj+bL8491Ad+Aydz/J3Se4+xGEYG034KbqbmBmI4GrCPNu7uXu57j7DwmTvf8BGA38NOmaAuDPhNWFTnX3b7n7T4H9gKeja65IuqYDcB+wHTjM3c939x8DI4A3gFPN7MxafAf1R5lLERERyZFMgst7gVcIyz9+bGbTzeyvZjYd+Bg4ITqe0fKP0dRFYwn9Oe9KOvwLYCNwtpm1reZWJwMG/MndF8Y73d2BKwnN9RclNVsfCuwOvO7uf0+4pgz4SfR2vJlZwjWnAt2Ax919ZsI1m4Gro7cXVVPXhqXMpYiIiORIjYPLKOA6BrgF2EbICH4jKrcCNwPH1mKlnnh0+UvJ17r7BmAaIbO4f/KFSeJs6sLkA9F9VhGyo8NSPHtSimsWAh8B/YABNbmG0FS+CTgwVXN63lDmUkRERHIkk8wl7r7N3a8EugB7AgdFZVd3vxrYHvVHzMRuUflRmuPzo3JINfdZFZW7Jh8ws/ZA1+jt0Do+O+017v4F8Alh5aP8nUx+wwZo3hxa5m/8KyIiIo1TRsFlzN3L3H22u09399nALtFgmsWEATWZ6BiV69Mcj/d3quY+z0flBWbWP+nYjYQmc4DE0ee1eXad6mtm48xsppnNXLlyZZpb5Fi8rniF1n4RERGRuqvV2uLw5WCYE4FxwNcIgaoTRlxnUxwBVTnFkbtPN7M/At8F3jOzp4E1hEE5+wAfAnsQBuJk9dmZXOPu9xL6rzJq1KhaTdtUZxs2qL+liIiI5ETGwWU0AOcC4BygR7R7FfBH4H53L8rwlnGmr2Oa4x2SzkvL3ceb2VuEgPf0aPcs4EjgfEJwuaKOz85afRtMnLkUERERybIaBZdm1pwwGnsccDghS7mV0AT+DeA5d7+mlnWYF5Xp+lQOjsp0/SIrcPcHgAeS95vZn6KXb9fx2fOAUdE1s5Ke0ZzQ5/MLUgwsyhvKXIqIiEiOVNnn0swGm9mvgWWEycHHAO8ClwE7u/tpWajDa1E51swq1CcaiDMaKAVm1PYBZjaWMOp7irsvSzj0alQeleKaAYQAsoiKgWLaawiTqrcBprv7ltrWN+eUuRQREZEcqW5Azzzgh0AZYULzYe4+yt3/4O5rslEBd18AvAT0By5JOnwd0BZ42N03xjvNbKiZDU06N57gPHnfQEIfx+3AhKTDU4A5wCFmdkLCNc2AX0Vv74nmyow9RegGcKaZjUq4phVh4BDAxHSfNy9s2KDgUkRERHKiJs3iDrwAPOXuH+aoHhcD04E7zGwMIeDbj9AE/xFh5Z1Ec6Iyebjz/WbWj9BcvRYYBBwPFAIXuHuF7Ke7bzezcwnZyKfM7CnCiPcxhKbvaYSgOvGaYjO7kBBkTjazxwkDh04gTFP0FPBEbb6EelNcrGZxERERyYnqMpc/JzQLnwtMM7PZZvYTM+uVzUpE2ctRwIOEoPKHwEDgDuAAd19dw1s9T5jg/XTgR8CBhGUcv+ruD6Z59puE0eTPEVYKuoIwWOd64Oupmrfd/VnC6j6vE/qcfi967g+AM5MynflHmUsRERHJkSozl+5+E3CTmR0JXEjIAt4S7XsJeChbFXH3JYQgtibnppyg0d0fqk2dork6M+o/6u7TCCsWNS5lZRrQIyIiIjlTo0nU3f1f7n4qsAthne4i4Gjg/wjN5iPMbO+c1VKyZ2PUdVWZSxEREcmBTJd/XOHut7j7IODrhP6F2whN2m+Z2TtmljwoR/JJvK64MpciIiKSA7Va/hHA3V9x9zOAPsBPCANv9iL0k5R8tWFDKJW5FBERkRyodXAZc/dV7n6ru+8OHEFoKpd8pcyliIiI5FCt1xZPxd0nA5OzeU/JMmUuRUREJIfqnLmURkaZSxEREckhBZdNjTKXIiIikkMKLpsaZS5FREQkhxRcNjXKXIqIiEgOKbhsaoqLobAQWrZs6JqIiIjIDkjBZVNTXKyspYiIiOSMgsumRuuKi4iISA4puGxqlLkUERGRHFJw2dQocykiIiI5pOCyqVHmUkRERHJIwWVTo8yliIiI5JCCy6ZGmUsRERHJIQWXTY0ylyIiIpJDCi6bkrIyKClR5lJERERyRsFlU1JSEkplLkVERCRHFFw2JcXFoVTmUkRERHJEwWVTsmFDKJW5FBERkRxRcNmUKHMpIiIiOabgsilR5lJERERyTMFlU6LMpYiIiOSYgsumRJlLERERyTEFl02JMpciIiKSYwoumxJlLkVERCTHFFw2JcXF0KIFtGzZ0DURERGRHZSCy6ZE64qLiIhIjim4bEqKi9XfUkRERHJKwWVTosyliIiI5JiCy6ZEmUsRERHJsbwJLs2sj5k9YGbLzWyLmS0ys9vNrHOG9znIzJ6Lrt9sZrZPOZ0AABfvSURBVIvN7AUzOyrFudeamVezLUi65rBqzr+lrt9FzihzKSIiIjnWvKErAGBmA4HpQHfgOWAusC/wfeAoMxvt7qtrcJ+LgLuBjcAzwFKgD3AKcLSZXe3uNyVcMrmK2x0PfBV4Mc3xKWmun1pdPRtMcTEMHNjQtRAREZEdWF4El4SAsDtwmbvfGe80s98CVwA3AeOruoGZFQK/BDYDe7v7vIRjNwPvAFeZ2a3uvgXA3SeTIkA0swLg/OjtvWkeOdndr63BZ8sfylyKiIhIjjV4s7iZDQDGAouAu5IO/4KQhTzbzNpWc6udgI7AR4mBJYC7zwE+AloD7WpQrWMIGc8Z7v5eDc5vHNTnUkRERHKswYNL4IiofMndyxIPuPsGYBrQBti/mvusAFYCQ8xscOIBMxsCDAberUnzOjAuKtNlLQEGmdmlZnalmZ2X/My8s307bNyozKWIiIjkVD40i+8WlR+lOT6fkNkcAryS7ibu7mZ2CfAoMMvMngGWA72Bk4EPgTOrq4yZ9QaOBtYDT1Rx6rejLfHap4EL3X1tdc+pdyUloVTmUkRERHIoH4LLjlG5Ps3xeH+n6m7k7n81s+XA/wHfSTj0OfBnYGEN6nMBUAA86u6bUhxfCUwA/kloym8FjAJuBr4B9DSzQ5KzsDEzG0eUGe3bt28NqpMlWldcRERE6kE+NItXx6LSqz3R7CzgZeA/wO6E5vTdCRnPPwCPV3N9M+C86G3KJnF3/9Ddf+XuH7h7ibuvcvdJwGHAJ8BowkjzlNz9Xncf5e6junXrVt1Hyp7i4lAqcykiIiI5lA/BZZyZ7JjmeIek81KK+lU+QGj+Ptvd57p7qbvPBc4GZgGnmdlhVdzmaKAvtRjI4+7FwF+it4dkcm29UOZSRERE6kE+BJfxyO4haY7HA2XS9cmMjQUKgSkpBgaVAa9Hb/eu4h7xQJ4/VvOsdFZGZXUj2+ufMpciIiJSD/IhuHwtKsdGzdJfMrP2hGbmUmBGNfdpGZXp2prj/VtTHTSznYFjCRnSJ6t5VjrxiPaa9O2sX8pcioiISD1o8ODS3RcALwH9gUuSDl9HyAI+7O4b451mNtTM/n979x4tWVneefz7E+kGFRsUECOOiAFhNN4CQoBREAchCRFMM2YyXEScGbzE0SWZcSEZGjM6uOKFgDFMTIAYjHgLGjOiRkGuRiQBJcpFkSYgKAjIaWgugzzzx96VVGqd6lOnqdrnUPX9rFXrXfXud+966+k6VU8/+7bLwNiL23Z1khf0L0jyImA1zXGb5w+ZyjE0J/L8xZATeXrb2nswCW77DwdeS5O8bmxyOjlWLiVJUgeWw9niAG+iuf3jqUn2B64B9gD2o9kd/q6B8de0be9kH6rq8iRnAkcD32ovRXQTTdJ6CLACOKWqvjv44m2yuNAdeXo+DjwuyWU0t5fcDNid5naVDwP/tarWLvyWO2blUpIkdWBZJJdVdUOS3YB3AwfS3CHnNuBU4KSqumvETR1Dc2zl64BXAVsAczT3+/5oVQ07W/xVwLNoTuS5eoHX+GPglTS767emSXB/BJxFk7x+e8S5dqtXuTS5lCRJE7QskkuAqrqZpuo4ytgM6S+aJO+sRb72efRVQRcY+z7gfYvZ/rKwbh2sWAErVy48VpIkaSMt+TGX6oj3FZckSR0wuZwVc3PuEpckSRNncjkr1q2zcilJkibO5HJWWLmUJEkdMLmcFVYuJUlSB0wuZ4WVS0mS1AGTy1lh5VKSJHXA5HJWWLmUJEkdMLmcBT//Oaxfb+VSkiRNnMnlLPC+4pIkqSMml7Ogl1xauZQkSRNmcjkL5uaa1sqlJEmaMJPLWWDlUpIkdcTkchZYuZQkSR0xuZwFVi4lSVJHTC5ngZVLSZLUEZPLWWDlUpIkdcTkchZYuZQkSR0xuZwF69bBypWwYsVSz0SSJE05k8tZ4H3FJUlSR0wuZ8G6dR5vKUmSOmFyOQusXEqSpI6YXM4CK5eSJKkjJpezwMqlJEnqiMnlLLByKUmSOmJyOQvm5kwuJUlSJ0wuZ8G6de4WlyRJnTC5nHYPPwzr11u5lCRJnTC5nHb33tu0Vi4lSVIHTC6nXe++4lYuJUlSB0wup926dU1r5VKSJHXA5HLaWbmUJEkdMrmcdlYuJUlSh5ZNcplk+yRnJLk1yYNJ1iY5JclWi9zOPkk+367/QJJ/SvLFJAcOGV8bePzdBl7n15N8Pck9Se5N8s0kRy32fU+clUtJktShxy/1BACSPAe4DNgW+DxwLfBS4L8BBybZu6ruHGE7bwQ+AtwHnAvcAmwPvAY4KMkJVfWeeVa9CThrnv5bhrzOW4DTgDuBs4GHgNXAWUl+qaqOW2iunbFyKUmSOrQskkuahHBb4K1VdVqvM8kHgbcD7wGO3dAGkmwK/G/gAeCXq+q6vmXvBa4E3pXk/VX14MDqa6tqzSgTTbID8H7gLmC3qlrb9r8b+BbwjiSfrapvjLK9ibNyKUmSOrTku8WT7AgcAKwF/mhg8Yk0VcgjkjxxgU09BVgFXN+fWAJU1TXA9cDmwJMe5ZRfD6wEPtxLLNvXuBt4b/t0g4lwp6xcSpKkDi15cgm8om2/UlWP9C+oqnXApcATgD0X2M7twB3Azkl26l+QZGdgJ+CqIbvXt0zy+iTHJ3lzkg29Vm++X5pn2XkDY5be3BxsthlsuulSz0SSJM2A5bBb/Llte/2Q5d+nqWzuDHxt2EaqqpK8meYYyL9Pci5wK/AM4FDgu8BvDVn9hcCf9Xck+TZwRFVdPep8q+q2JPcB2yd5QlWtHzbfzszNWbWUJEmdWQ6Vy1Vte8+Q5b3+LRfaUFV9mqZq+DPgSOCdwBE0u9bPBH44z2ofBPYGtgG2AHYHPkOTcJ6f5BkbOd9V8y1M8l+SXJHkijvuuGOht/TorVvn8ZaSJKkzyyG5XEjathYcmBwOfBW4GNiVZnf6rjQVzw8D5wyuU1XvqKrLquqnVXVvVV1RVYcBnwW2BhZ75vcG51tVf1JVu1XVbttss80iN70RrFxKkqQOLYfkcoOVPuDJA+Pm1R5XeQbN7u8jquraqrq/qq6lqV7+PXBYkn1HnNfpbfuyjZzv3IivM1lWLiVJUoeWQ3LZO7N75yHLeyfnDDsms+cAYFPgwnlODHoEuKh9+ssjzqu3z3rwLPWh803y9Hb8LcvieEuwcilJkjq1HJLLC9r2gCT/aj5JtqA5HvJ+YOjdclor23bYvuZe/0Mjzqt3xvjgcZrnt+18d/w5aGDM0rNyKUmSOrTkyWVV3QB8BdgBePPA4pNoKoEfq6r7ep1Jdkmyy8DYi9t2dZIX9C9I8iKaO+gUfYlfkpfMd/3Mdv3enXzOHlh8JvAg8Jb2guq9dbYCjm+fns5yYeVSkiR1aDlcigjgTTS3fzw1yf7ANcAewH40u8PfNTD+mrbtnTxDVV2e5EzgaOBb7aWIbqJJWg8BVgCnVNV3+7bzVuA1Sc4HbqZJGnehqUpuAnwU+ET/C1fVjUl+FzgVuCLJJ/mX2z9uD3xg2dydB6xcSpKkTi2L5LKqbkiyG/BumsTuV4HbaBK4k6rqrhE3dQzNsZWvA15Fc2mhOeAS4KNVNXi2+OdoTsB5Ac0ljDajuV/4ee34vx4y39OSrKU5k/xImgrw94ATqurPR5zr5D38MNx/v5VLSZLUmWWRXAJU1c00VcdRxmZIfwFntY9RtvM5mgRz0arqC8AXNmbdzvRu/WjlUpIkdWTJj7nUBM21V0OycilJkjpicjnNrFxKkqSOmVxOMyuXkiSpYyaX08zKpSRJ6pjJ5TSzcilJkjpmcjnNrFxKkqSOmVxOMyuXkiSpYyaX06xXuTS5lCRJHTG5nGZzc7DZZrDppks9E0mSNCNMLqfZjjvCQQct9SwkSdIMMbmcZm98I/zVXy31LCRJ0gwxuZQkSdLYmFxKkiRpbEwuJUmSNDYml5IkSRobk0tJkiSNjcmlJEmSxsbkUpIkSWNjcilJkqSxMbmUJEnS2JhcSpIkaWxMLiVJkjQ2JpeSJEkaG5NLSZIkjU2qaqnnMLOS3AHcNOGX2Rr46YRfYxYZ18kwrpNhXCfDuE6GcZ2MccT1WVW1zUKDTC6nXJIrqmq3pZ7HtDGuk2FcJ8O4ToZxnQzjOhldxtXd4pIkSRobk0tJkiSNjcnl9PuTpZ7AlDKuk2FcJ8O4ToZxnQzjOhmdxdVjLiVJkjQ2Vi4lSZI0NiaXkiRJGhuTyymUZPskZyS5NcmDSdYmOSXJVks9t+UuyeokpyW5OMlckkpy9gLr7JXki0nuSrI+yXeSvC3JJl3NezlL8tQkb0hybpIfJLk/yT1JLklyTJJ5v4eM68KSvC/J15Lc3Mb1riRXJjkxyVOHrGNcFynJEe13QSV5w5Axv57k6+1n+94k30xyVNdzXc7a36Ia8vjxkHX8vI4oyb9L8tkkt7W//bcl+UqSX51n7ETj6jGXUybJc4DLgG2BzwPXAi8F9gOuA/auqjuXbobLW5KrgBcC9wK3ALsAH6+qw4eMfzXwWeAB4JPAXcDBwHOBz1TVYV3MezlLcizwx8BtwAXAPwFPA14DrKKJ32HV92VkXEeT5CHgH4DvAbcDTwT2BHYDbgX2rKqb+8Yb10VK8kzgamAT4EnAf66qPx0Y8xbgNOBOmrg+BKwGtgc+UFXHdTrpZSrJWmBL4JR5Ft9bVe8fGO/ndURJTgB+n+Yi6X9D8327NfBi4IKq+u99Yycf16ryMUUP4MtAAb8z0P/Btv/0pZ7jcn7QJOE7AQH2bWN29pCxT6b5QX8Q2K2vfzOaBL+A31rq97TUD+AV7RfX4wb6t6NJNAv4TeO6UbHdbEj/e9o4fcS4Pqr4BvgqcAPwB22M3jAwZgeaH+k7gR36+rcCftCu8ytL/V6WwwNYC6wdcayf19Hjelgbj78Ftphn+aZdx9Xd4lMkyY7AATR/wH80sPhE4D7giCRP7HhqjxlVdUFVfb/av7YFrAa2Ac6pqiv6tvEAcEL79I0TmOZjSlWdX1VfqKpHBvp/DJzePt23b5FxHVEbk/l8qm136uszrov3Vpr/HB1N8/05n9cDK4EPV9XaXmdV3Q28t3167ATnOK38vI6gPazofcB64Lerat3gmKr6f31PO4nr4x/tBrSsvKJtvzLPD/m6JJfSJJ97Al/renJTqBfvL82z7CKaP/a9kqysqge7m9ZjSu9L7+G+PuP66B3ctt/p6zOui5BkV+Bk4A+r6qIkrxgydENxPW9gjGBlksOBf0OTsH8HuKiqfj4wzs/raPYCng18Brg7ya8Bz6eppl9eVd8YGN9JXE0up8tz2/b6Icu/T5Nc7ozJ5TgMjXdVPZzkRuB5wI7ANV1O7LEgyeOBI9un/V90xnWRkhxHczzgKprjLfeh+dE+uW+YcR1R+9n8C5rDNo5fYPiG4npbkvuA7ZM8oarWj3emj0nb0cS2341Jjq6qC/v6/LyOZve2/QnN8de/1L8wyUXA6qq6o+3qJK7uFp8uq9r2niHLe/1bdjCXWWC8H52Taf6H/cWq+nJfv3FdvONoDn15G01i+SXggL4fFDCui/E/aU6EeF1V3b/A2FHjumrI8llyJrA/TYL5RJpE6P/QHLd6XpIX9o318zqabdv2WGBz4JXAFjTfrV8GXgZ8um98J3E1uZwtaVsvEdAN4z1EkrcC76C5msERi129bY1rq6q2q6rQ/Gi/hqbqcGWSlyxiM8YVSPJSmmrlB+bZpbhRm2zbmY4rQFWd1B6D/ZOqWl9V/1hVx9KccLo5sGYRmzOujd6lg0JTofxaVd1bVd8FDqW56snLk/zKiNsbS1xNLqfLQv9DfvLAOD06xnsjJHkz8Ic0l8/Zr6ruGhhiXDdS+6N9Ls3hL08FPta32LguoG93+PXA74242qhxnXsUU5t2vRP7XtbX5+d1NHe37Q+r6tv9C9qqe2+v0EvbtpO4mlxOl+vaduchy3tnjg47JlOLMzTe7Y/Us2lOVPlhl5NazpK8Dfgw8I80ieV8F042ro9SVd1Ek7w/L8nWbbdxXdiTaOKzK/BA/0W+aQ47APho29e7VuOG4vp0mt2/t3i85Qbd3rb9VzLx8zqaXpx+NmR5L/ncfGD8RONqcjldLmjbAwbvepJkC2Bv4H7g77qe2JQ6v20PnGfZy4AnAJfN+JmM/yzJ/wA+BFxFk1jePmSocR2PX2jb3lm4xnVhDwJ/NuRxZTvmkvZ5b5f5huJ60MAYza+3y7Y/ofHzOpqLaJLBnZKsmGf589t2bdt2E9elvvinj/E+8CLq44zlvix8EfU78CK/o8Ty99p4XAE8ZYGxxnW0mO4CbDdP/+P4l4uoX2pcxxbvNcx/EfVn40XUR4nf8+b72weeRXMlkwKO7+v38zp6bM9u4/G/Bvr/PfAITVVzyy7j6u0fp8w8t3+8BtiD5s4z1wN7lbd/HCrJIcAh7dPtgFfR/G/64rbvp9V3K7d2/GdoflzOobmN1m/Q3kYL+A81439k7f2Vz6KpoJ3G/MfyrK2qs/rWMa4LaA8x+AOaysUNNMnN04CX05zQ82Ng/6r6Xt86xnUjJVlDs2t8vts//g5wKt7+cag2fu+k2cN2I7AOeA7wazSJzReBQ6vqob51/LyOIMm2wKXAL9L8Vl1Ok7QfSpMs/nZVfbpv/OTjutQZt4/xP4Bn0lzy4TaaL7mbaE6g2GDFyMe/qk4Me6ydZ529ab4Y76Y57OBq4O3AJkv9fpbDY4SYFvB147rouD6f5k5cV9HcT/hhmsT9W23M5/17N64bHe/e5/gNQ5YfDFxIkzTd1/47HLXU814uD5r/9HyC5goRP6O5gcIdNLcsPBKaYtc86/l5HS2+T6HZQ3lj+7t/J02Bac+liKuVS0mSJI2NJ/RIkiRpbEwuJUmSNDYml5IkSRobk0tJkiSNjcmlJEmSxsbkUpIkSWNjcilJkqSxMbmUJA2VZE2SSrLvUs9F0mODyaUkTVCbmC302Hep5ylJ4/L4pZ6AJM2IkzawbG1Xk5CkSTO5lKQOVNWapZ6DJHXB3eKStIz0H+OY5KgkVya5P8ntSc5Ist2Q9XZK8rEkP0ryUJJb2+c7DRm/SZJjk1ya5J72NX6Q5E83sM7qJJcnWZ/kriTnJHnGON+/pMc+K5eStDy9HTgA+CTwJWAf4Ghg3yR7VNUdvYFJdge+CmwB/DXwPWAX4D8Br06yf1Vd0Td+BfB/gVcCNwN/CcwBOwCHApcA3x+Yz5uA32i3fyGwB/Ba4IVJXlRVD47zzUt67DK5lKQOJFkzZNEDVXXyPP0HAXtU1ZV92/gQ8DbgZOCYti/Ax4AnA4dX1cf7xr8WOAc4O8m/rapH2kVraBLLLwCH9SeGSVa22xp0ILB7VV3dN/Yvgf8IvBr41NA3L2mmpKqWeg6SNLWSLPQle09Vbdk3fg1wInBGVR0zsK1VwE3ASmDLqnowyd40lcZvVNVe87z+xTRVz5dX1UVJNgHuBFYAv1hVty4w/9583lNVJwws2w84H/hAVR23wPuUNCM85lKSOlBVGfLYcsgqF86zjXuAq4DNgF3b7pe07flDttPrf3Hb7gKsAr6zUGI54Ip5+m5u260WsR1JU87kUpKWp58M6f9x264aaG8bMr7Xv+VA+6NFzudn8/Q93LabLHJbkqaYyaUkLU9PG9LfO1v8noF23rPIgacPjOsliZ7lLWkiTC4laXl6+WBHe8zli4AHgGva7t4JP/sO2U6v/x/a9lqaBPMFSX5hHBOVpH4ml5K0PB2R5MUDfWtodoN/ou8M70uB64B9kqzuH9w+fxlwPc1JP1TVz4GPAJsDp7dnh/evsyLJNmN+L5JmiJcikqQObOBSRACfq6qrBvrOAy5N8ima4yb3aR9rgXf2BlVVJTkK+Fvgk0k+T1OdfC5wCLAOOLLvMkTQ3IpyD+Bg4Pokf9OOeybNtTV/Fzhro96opJlncilJ3ThxA8vW0pwF3u9DwLk017V8LXAvTcJ3fFXd3j+wqr7ZXkj9BJrrVx4M/BT4BPD7VXXdwPiHkhwIHAscCRwFBLi1fc1LFv/2JKnhdS4laRnpu67kflX19aWdjSQtnsdcSpIkaWxMLiVJkjQ2JpeSJEkaG4+5lCRJ0thYuZQkSdLYmFxKkiRpbEwuJUmSNDYml5IkSRobk0tJkiSNjcmlJEmSxub/A2YTqY7nb88xAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(1, 1, figsize=(10,6))\n",
    "ax.plot(np.sqrt(model_history.history['acc']), 'r', label='train_acc')\n",
    "ax.plot(np.sqrt(model_history.history['val_acc']), 'b' ,label='val_acc')\n",
    "ax.set_xlabel(r'Epoch', fontsize=20)\n",
    "ax.set_ylabel(r'Accuracy', fontsize=20)\n",
    "ax.legend()\n",
    "ax.tick_params(labelsize=20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAn0AAAGJCAYAAAD/mIVfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3XeYVOXd//H3d2GBZWm7FEFQaVKlqIuioAIag1hijxo1mhgfTYzliU9i+0VNYmKKPVGjiVFji11jxQIqimVJwEIVBUGR3mFpe//++M64wzLDtmk783ld17nO7Dlnzt5zXHc/3NVCCIiIiIhIbivIdAFEREREJPUU+kRERETygEKfiIiISB5Q6BMRERHJAwp9IiIiInlAoU9EREQkDyj0iYiIiOQBhT4RERGRPKDQJyIiIpIHFPpERERE8kDTTBcgG3Xo0CF0794908UQERERqdGUKVOWhRA61nSdQl8c3bt3p7y8PNPFEBEREamRmc2vzXVq3hURERHJAwp9IiIiInlAoU9EREQkD6hPn4iIiDRqW7ZsYeHChVRUVGS6KCnVokULunXrRmFhYb3er9AnIiIijdrChQtp3bo13bt3x8wyXZyUCCGwfPlyFi5cSI8ePep1DzXvioiISKNWUVFB+/btczbwAZgZ7du3b1BtpkKfiIiINHq5HPiiGvoZFfpEREREGmDVqlXcfvvtdX7fuHHjWLVqVQpKFJ9Cn4iIiEgDJAp927Zt2+n7XnjhBdq1a5eqYu1AAzlEREREGuCyyy5j7ty5DB06lMLCQlq1akWXLl2YOnUq06dP59hjj2XBggVUVFRw0UUXce655wJVK4CtW7eOI444gpEjR/LOO+/QtWtXnnnmGYqKipJaToU+ERERyR0XXwxTpyb3nkOHws03Jzx9/fXX8/HHHzN16lQmTpzIkUceyccff/zNKNt77rmH0tJSNm7cyLBhwzjhhBNo3779dveYM2cODz/8MHfffTcnn3wyTzzxBKeffnpSP4aadzNh8WJ4/nlYuzbTJREREZEk22+//babVuXWW29lyJAhDB8+nAULFjBnzpwd3tOjRw+GDh0KwL777su8efOSXi7V9GXC5Mlw3HHwn//A3ntnujQiIiK5Yyc1culSXFz8zeuJEyfy6quvMnnyZFq2bMmoUaPiTrvSvHnzb143adKEjRs3Jr1cqunLhNJS369YkdlyiIiISIO1bt2atQla71avXk1JSQktW7Zk5syZvPvuu2kuXRXV9GWCQp+IiEjOaN++PSNGjGCvvfaiqKiIXXbZ5ZtzY8eO5c4772Tw4MH07duX4cOHZ6ycCn2ZoNAnIiKSUx566KG4x5s3b86LL74Y91y0316HDh34+OOPvzl+6aWXJr18oObdzCgp8b1Cn4iIiKSJQl8mFBX5ptAnIiIiaaLQlymlpQp9IiIikjYKfZmi0CciIiJppNCXKQp9IiIikkYKfZmi0CciIiJppNCXKQp9IiIiealVq1YZ+b4KfZmi0CciIiJppMmZM6W0FCoqYONGn75FREREGqVf/OIX7LHHHvz4xz8G4JprrsHMePPNN1m5ciVbtmzhN7/5Dd/5zncyWk6FvkyJXZWja9fMlkVERCRHXHwxTJ2a3HsOHQo335z4/CmnnMLFF1/8Teh79NFHeemll7jkkkto06YNy5YtY/jw4RxzzDGYWXILVwcKfZmi0CciIpIT9t57b5YsWcJXX33F0qVLKSkpoUuXLlxyySW8+eabFBQU8OWXX7J48WI6d+6csXJmPPSZ2YnAIcBQYAjQGngwhHB6Pe7VDfgVMBZoDywCngauDSGsTFqhk0Hr74qIiCTdzmrkUunEE0/k8ccf5+uvv+aUU07hwQcfZOnSpUyZMoXCwkK6d+9ORUVFZgoXkfHQB1yFh711wEKgX31uYma9gHeATsAzwExgP+AiYKyZjQghLE9KiZNBoU9ERCRnnHLKKfzoRz9i2bJlvPHGGzz66KN06tSJwsJCJkyYwPz58zNdxKwYvXsJ0AdoA5zfgPvcjge+C0MIx4YQLgshjAFuAvoC1zW4pMmk0CciIpIzBg4cyNq1a+natStdunThe9/7HuXl5ZSVlfHggw/Sr1+96rSSKuM1fSGECdHX9e3caGY9gcOBecBfqp2+GjgXOMPMfhZCWF+/kiaZQp+IiEhO+eijj7553aFDByZPnhz3unXr1qWrSNvJhpq+ZBgT2Y8PIVTGngghrAXeBloCw9NdsISKi6GwUKFPRERE0iJXQl/fyH52gvNzIvs+aShL7ZhpgmYRERFJm1wJfW0j+9UJzkePt0tDWWpPoU9ERETSJFdCX02inQVDwgvMzjWzcjMrX7p0aXpKpdAnIiKSFCEk/BOfMxr6GXMl9EVr8tomON+m2nU7CCHcFUIoCyGUdezYMamFS0ihT0REpMFatGjB8uXLczr4hRBYvnw5LVq0qPc9Mj56N0lmRfaJ+uztGdkn6vOXGaWlMG1apkshIiLSqHXr1o2FCxeStpa6DGnRogXdunWr9/tzJfRFp3053MwKYkfwmllrYASwEXg3E4VLSDV9IiIiDVZYWEiPHj0yXYys16iad82s0Mz6RVbf+EYIYS4wHugO/KTa264FioH7s2aOvqjSUli3DjZvznRJREREJMdlvKbPzI4Fjo18GV2F+AAzuzfyelkI4dLI667ADGA+HvBi/Rhfhu1WMzs0ct3+wGi8WffKVJS/QaITNK9cCbvsktmyiIiISE7LeOgDhgLfr3asZ2QDD3iXUoMQwlwzKwN+BYwFxgGLgFuBa0MI2deO2r6971esUOgTERGRlMp46AshXANcU8tr51E1/Uq88wuAs5NRrrTQUmwiIiKSJo2qT1/OUegTERGRNFHoyySFPhEREUkThb5MUugTERGRNFHoy6Q2baBJE4U+ERERSTmFvkwyg5IShT4RERFJOYW+TNOqHCIiIpIGCn2ZVloKy5dnuhQiIiKS4xT6Mk01fSIiIpIGCn2ZptAnIiIiaaDQl2kKfSIiIpIGCn2ZVloKq1fD1q2ZLomIiIjkMIW+TItO0LxqVWbLISIiIjlNoS/TtCqHiIiIpIFCX6Yp9ImIiEgaKPRlmkKfiIiIpIFCX6Yp9ImIiEgaKPRlmkKfiIiIpIFCX6a1a+d7hT4RERFJIYW+TGvSxIOfQp+IiIikkEJfNtCqHCIiIpJiCn3ZQKFPREREUkyhLxso9ImIiEiKKfRlA4U+ERERSTGFvmyg0CciIiIpptCXDUpLYeVKqKzMdElEREQkRyn0ZYPSUg98a9ZkuiQiIiKSoxT6soFW5RAREZEUU+jLBgp9IiIikmIKfdlAoU9ERERSTKEvGyj0iYiISIop9GUDhT4RERFJMYW+bKDQJyIiIimm0JcNCguhdWuFPhEREUkZhb5soVU5REREJIUU+rKFQp+IiIikkEJftlDoExERkRRS6MsWCn0iIiKSQgp92UKhT0RERFJIoS9bRENfCJkuiYiIiOQghb5sUVoKW7bA+vWZLomIiIjkIIW+bKEJmkVERCSFFPqyhUKfiIiIpJBCX7ZQ6BMREZEUUujLFgp9IiIikkIKfdlCoU9ERERSSKEvW5SU+H758syWQ0RERHKSQl+2KCryTTV9IiIikgIKfdlEq3KIiIhIiij0ZROFPhEREUkRhb5sotAnIiIiKaLQl00U+kRERCRFsib0mVk3M7vHzL4ys01mNs/MbjazkjreZ6SZPRN5f4WZfWFmL5jZ2FSVPWkU+kRERCRFsiL0mVkvYApwNvA+cBPwGXARMNnM2tfyPucDbwGHRvY3AW8AhwAvmtmVyS99Ein0iYiISIo0zXQBIm4HOgEXhhBuix40sxuBS4DrgPN2dgMzKwR+B1QA+4YQZsWc+y3wX+BKM/tTCGFT8j9CEpSWQkUFbNzo07eIiIiIJEnGa/rMrCdwODAP+Eu101cD64EzzKy4hluVAm2B2bGBDyCEMAOYDRQBrZJQ7NTQqhwiIiKSIhkPfcCYyH58CKEy9kQIYS3wNtASGF7DfZYAS4E+ZrZn7Akz6wPsCUwNIWTvkhcKfSIiIpIi2RD6+kb2sxOcnxPZ99nZTUIIAfgJ/pmmmNl9ZvY7M7sf7y/4CXBSEsqbOgp9IiIikiLZ0KevbWS/OsH56PF2Nd0ohPCYmX0FPAycGXNqMfAPfHBI9lLoExERkRTJhpq+mlhkH2q80Ox04FV85G5/vFm4P/Aa8GfgkZ2891wzKzez8qVLlza40PWi0CciIiIpkg2hL1qT1zbB+TbVrosr0m/vHrwZ94wQwswQwsYQwkzgDLyJ9yQzGxXv/SGEu0IIZSGEso4dO9b1MySHQp+IiIikSDaEvuhI20R99qKDMhL1+Ys6HCgE3ogzIKQSeDPy5b71KWRaFBdDYaFCn4iIiCRdNoS+CZH94Wa2XXnMrDUwAtgIvFvDfZpH9omq6aLHN9enkGlhpgmaRUREJCUyHvpCCHOB8UB3fPRtrGuBYuD+EML66EEz62dm/apd+1Zkf6KZDY49YWZDgRPxfoGvJ6/0KaDQJyIiIimQDaN3AX4MvAPcamaHAjOA/YHReLNu9eXTZkT20UEehBDeN7N/4Eu5fWBmTwHz8TB5LNAMuDmE8EkKP0fDKfSJiIhICmRF6AshzDWzMuBXwFhgHLAIuBW4NoRQ2xT0Q7zv3lnAt4HWwBpgEnB3CCHh6N2s0b49fPFFpkshIiIiOSYrQh9ACGEBXktXm2stwfEA3BvZGqfSUpg6NdOlEBERkRyT8T59Uo2ad0VERCQFFPqyTWkprFsHm7N3kLGIiIg0Pgp92SY6QfPKlZkth4iIiOQUhb5so1U5REREJAUU+rKNQp+IiIikgEJftlHoExERkRRQ6Ms2Cn0iIiKSAgp9GTB3LvzpT7B2bZyTCn0iIiKSAgp9GTBjBvzf/8GHH8Y52aYNNGmi0CciIiJJpdCXAUOG+D5u6DODkhKFPhEREUkqhb4M6NYN2rVLEPpAq3KIiIhI0in0ZYAZDB4M06YluEChT0RERJJMoS9DBg+Gjz6Cyso4JxX6REREJMkU+jJk8GBfYnfevDgnFfpEREQkyRT6MmTwYN/H7den0CciIiJJptCXIXvt5X37Eoa+Vatg27a0l0tERERyk0JfhhQXQ+/eOwl94MFPREREJAkU+jIo4QhercohIiIiSabQl0GDB/uSbOvWVTsRDX3Ll6e9TCIiIpKbFPoyaPBgCAE++aTaCdX0iYiISJIp9GVQwhG8Cn0iIiKSZAp9GdS9O7RqpdAnIiIiqafQl0EFBV7bt0Poa9fO9wp9IiIikiQKfRkWHcEbQszBJk08+Cn0iYiISJIo9GXY4MGwejUsWFDthFblEBERkSRS6MuwnQ7mUOgTERGRJFHoy7C99vL9DqGvY0dYuDDt5REREZHcpNCXYW3b+ijeHULfyJHw0Ufw9deZKJaIiIjkGIW+LDBkSJzQN26c7196Ke3lERERkdyj0JcFBg+GWbNg48aYg0OGwK67wgsvZKxcIiIikjsU+rLA4MFQWQnTp8ccNPPavvHjYcuWjJVNREREcoNCXxZIOIJ33Difz2Xy5LSXSURERHKLQl8W6NULiorihL5DD4XCQnj++YyUS0RERHKHQl8WaNLEp27ZIfS1aQMHHaR+fSIiItJgSQ19ZlZiZsXJvGe+iLscG3gT78cfwxdfZKRcIiIikhvqHPrM7FAz+4OZlcQc62RmbwDLgBVmdmMyC5kPhgyB5cth0aJqJ4480vcvvpj2MomIiEjuqE9N30+B40MIK2OO/Qk4CPgUWA5cZGYnJ6F8eSPhYI6+faFHDzXxioiISIPUJ/QNASZFvzCzIuBE4JUQQl+gL7AAOC8pJcwTgwb5fofQF5265dVXYdOmtJdLREREckN9Ql8n4KuYr/cHWgD3AoQQ1gLP4eFPaqm0FLp1ixP6wEPfhg3wxhtpL5eIiIjkhvqEvk1AUczXBwEBeDPm2BqgtAHlykuDBycIfaNGQYsWauIVERGReqtP6PscGBPz9QnAnBDClzHHdsMHdUgdDB4MM2bA5s3VTrRsCaNHK/SJiIhIvdUn9N0HDDKz98zsLWAQ8FC1a/YBZjW0cPlmyBDYutWD3w6OPBLmzPFNREREpI7qE/ruAB4ByoAReP+930dPmtl+QH9gYhLKl1cSjuAFOOII32vqFhEREamHOoe+EMKWEMJpQAnQNoTwnRBC7LDSz4C9gduSVMa80acPNGuWIPT17An9+qmJV0REROql3ityhBDWREbqVj++LIQwLYSwumFFyz9Nm8LAgQlCH/go3okTYf36dBZLREREckB9VuQoMbMBZta82vGzzewZM3so0sQr9ZBwBC946Nu0CV5/Pa1lEhERkcavPjV9vwXei32vmf0U+BtwNHAKMNHMBiSlhHlm8GD4+mtYsiTOyZEjoVUrNfGKiIhIndUn9I0AXgshbIw5dinwJXAwEF1+7X8bWLa8tNPBHM2bw2GHeegLIa3lEhERkcatPqGvKz5XHwCRGr3dgNtCCJNCCI8D/8YDoNTRkCG+T9jEe+SR8MUXMH162sokIiIijV99Ql8RUBHz9Qh8RY5XY47NxcOh1FHHjtC5805CX3TqFjXxioiISB3UJ/R9CfSL+frb+LJr02KOlQCxzb9SBzsdzNG1q1cHPv98WsskIiIijVt9Qt8EYJyZXWBm5wDHAC+FECpjrukNLEhGAfPR4MHwySe+Okdc48bBpEmwWrPiiIiISO3UJ/T9DlgH3ALchTf1XhM9aWadgEOAd+pyUzPrZmb3mNlXZrbJzOaZ2c1mVlLXAprZIDO738wWRO61xMzeMLMz63qvTBg82NffnT07wQXjxsG2bfDKK2ktl4iIiDRe9VmR43NgIHARcCGwVwghdp3dPYC/APfW9p5m1guYApwNvA/chK/scREw2cza1+FeZwH/BY4F3gJuAB4HDBhX2/tkUnQE77RpCS4YPhxKStSvT0RERGqtaX3eFEL4GvhzgnMfAB/U8Za3A52AC0MI3yzfZmY3ApcA1wHn1XQTMxuOzxf4MTA2Us7Y84V1LFdG9O/vq3N8+CGcemqcC5o2hW9/29fhrayEgnovrCIiIiJ5okFpwcwKI02pB5nZ4PqEKjPrCRwOzMNrCGNdDawHzjCz4lrc7g9AE+D06oEPfN3gupYvE5o18+A3ZcpOLho3zmdx/u9/01YuERERabzqFfrMrI2Z3QmsAqYCE/Em1VVmdqeZtavD7cZE9uOrDQYhsrbv20BLYHgNZeoGHASUA5+Y2Wgzu9TMfmZmh5pZo6oOO/po77I3cWKCC444widr/nPcClcRERGR7dRn7d02eBA7F9iK95t7NLLfEjk+KXJdbfSN7BMNW5gT2fep4T7DYq5/PbL9EfgTPofgVDPrXcsyZdyVV0LPnnDuubAx3uQ3HTrABRfA/ff7UF8RERGRnahP7dfl+ECOO4A9QgijQginhhBGUTWIY0DkutpoG9knmn8kerym2sNOkf3JQH/g+Mi9ewP/BAYBz5tZs3hvNrNzzazczMqXLl1ay6KnTsuW8Ne/wpw58OtfJ7jo8st9Ld4rr0xr2URERKTxqU/oOx54N4TwkxDCqtgTIYTVIYSfApOBE5JRQHzULfiqHzvTJGZ/TgjhqRDCmhDCXOD7eLNvn0TlCiHcFUIoCyGUdezYMRnlbrDDDoOzzoI//CHBSN727eHnP4dnnoF36jRDjoiIiOSZ+oS+3fE+fDvzBr4eb21Ea/LaJjjfptp1iayM7DcB281lEkIIwDORL/erZbmywp/+5NnunHMSTNZ88cW+bttll0GoKReLiIhIvqpP6NtAVVNqIh0j19VGdI6/RH329ozsE/X5q36ftdUHhEREQ2FRLcuVFdq3h1tvhfJy3++guBh++Ut46y3N2yciIiIJ1Sf0fQCcZGZ7xjsZmWj5ZGo/V9+EyP7w6iNszaw1MAJfx/fdGu7zIbAM6GBmu8Q5v1dkP6+W5coaJ58MRx0FV10Fn30W54JzzoHevb2P37ZtaS+fiIiIZL/6hL4/Aq2AD8zs12Y2xsz6R6ZIuRYPe63wUbM1ivS5Gw90B35S7fS1QDFwfwhhffSgmfUzs37V7rMV+Gvkyz/EBkgzGwSchY82fry2HzRbmMHtt0OTJnDeeXFacQsL4Te/gY8+gocfzkgZRUREJLtZqEc/MDP7H3zt3eqTMRs+bcvFIYQ76nC/XvhavZ3wvnczgP2B0Xiz7oEhhOUx1weAEIJVu09L4DV8Tr//4n0PO+KDN4qAn4UQbqypPGVlZaG8vLy2xU+bv/zFZ2m57z44s/oqwpWVUFYGK1fCzJk+h5+IiIjkPDObEkIoq/G6+oS+yDfYHTgD2BsfhLEaD1oPhBDm1+N+uwG/AsYC7YFFwNPAtSGEFdWujRv6IudaAj8HTgF6ABV47eMNIYQXa1OWbA19lZUwciTMmgUzZkCn6j0rx4/35dluuQUuvDAjZRQREZH0Snnoq+GbtwCahRDWJP3maZCtoQ9g+nQYOhROPBEeeqjayRB8npePPoK5c6F164yUUURERNKntqEvVUuT3QGsqPEqqbMBA3wu5ocfhuefr3bSDK6/HpYuhRtuyEj5REREJDulcj3aHZpeJTkuu8zD3/nnw9q11U4OG+bVgDfcAEuWZKR8IiIikn1SGfokRZo3h7vvhgULvGJvB7/5jS/Ye911aS+biIiIZCeFvkbqwAPhu9+Fm2+GxYurnezbF37wA7jjDvj884yUT0RERLKLQl8j9qtfwaZNCSr0rr7aJ/b75S/TXi4RERHJPgp9jVifPnD22XDnnTC/+iQ5XbvCJZfAAw/AK69kpHwiIiKSPRT6Grlf/hIKCuCaa+Kc/H//D/r3h7POghUaTC0iIpLPahX6zGxbXTag+noRkiK77QY//jHcf79P2LydoiKv6VuyBH5SfYU7ERERySe1remzemySJpdfDi1besXeDvbZB669Fh55ROvyioiI5LFahb4QQkE9tiapLry4jh3hf/8XnngC4i4k8vOf+3Df88/3eV5EREQk76hPX4742c+gtNRX69hB06be/rt1q/fvq6xMd/FEREQkwxT6ckSbNt7MO348TJwY54JevXxSv9dfh1tvTXfxREREJMMshJDpMmSdsrKyUB63nTS7bdwIvXvDHnvA22/7UrzbCQGOPRZefhmmTIGBAzNSThEREUkeM5sSQiir6TrV9OWQoiKfwmXyZHj++TgXmPn6bW3awOmnw+bNaS+jiIiIZIZCX475wQ+8JffKKxN03evUCf72N5g61VftEBERkbyg0JdjCgt9ebYPP4R//SvBRcccA+ecA7//PUyalNbyiYiISGaoT18cjbVPX1RlJQwdChs2+ITNhYVxLlq71i+qrIRp07zJV0RERBod9enLYwUFcN11MHcu/OMfCS5q3dqncfniC/jRjzSNi4iISI5T6MtRRx0FBxzga/KuWpXgohEj4Le/hUcfTbB4r4iIiOQKhb4cZQa33FKLZXd//nMf/fHrX3vNn4iIiOQkhb4cNmyYT+Hy0EO+9G5cZnDHHTB6tA/uePPNtJZRRERE0kOhL8ddcQUMH17DsrvNmvnCvT17wnHHwZw5aS2jiIiIpJ5CX45r2hT++U/YsqWGZXdLSnxGZzM48khYsSKdxRQREZEUU+jLA717w003+bK7t9yykwt79YKnn4b58+H447Vih4iISA5R6MsT55zjczJffjl8/PFOLhw5Eu65B954A84919frFRERkUZPoS9PRJfdbdsWvvc92LRpJxd/73u+RNt998H116etjCIiIpI6Cn15pFMnr8T78EO46qoaLr76ajjtNB8J8thjaSmfiIiIpI5CX5458kj4n/+BG26AiRN3cqEZ/P3vPoHzmWfCO++kq4giIiKSAgp9eeiGG3xwx5ln7mS1DoAWLeCpp6BbNzj6aJg5M21lFBERkeRS6MtDxcXwwAPw1VdwwQU1XNyxI7z0ks/9MnYsLFqUljKKiIhIcin05an99vPVOh58EO66q4aLe/XyOfyWLYNx42DNmrSUUURERJJHoS+PXXEFHHaY9/G78MIapuUrK4PHH/f5Xk44QXP4iYiINDIKfXmsaVN44QW45BK47TYYM8abfBMaOxb+9jd49VX4wQ92sryHiIiIZBuFvjxXWAg33giPPAJTp8I++8Cbb+7kDd//Plx3nbcLX3FF2sopIiIiDaPQJwB897vw3ns+efOYMb5sW8LFOC6/HM4/H37/e68iFBERkayn0CffGDgQ3n/fZ2f53/+FU0+FdeviXGjmYe/YY+Gii7yvn4iIiGQ1hT7ZTtu28OSTvvraY4/B8OEwe3acC5s0gYceggMOgNNPr6FNWERERDJNoU92YAa/+AWMHw+LF8OwYQkW5CgqgmefhR49fCqXV19Ne1lFRESkdhT6JKFDD4UpU2CXXeCII/z1Dtq3h9dfh549PfipqVdERCQrKfTJTu2+O7z2GpSWwuGHw0cfxbmoSxd44w2f8fnkk+Gvf017OUVERGTnFPqkRrvt5sGvqMgnc541K85FJSXeHnzEEXDeefDb3+5k+K+IiIikm0Kf1ErPnh78wJt9P/sszkUtW8LTT/vAjiuvhJ/9TBM4i4iIZAmFPqm1vn19rMbGjT6X34IFcS4qLIT77vN13W66Cc46C7ZsSXdRRUREpBqFPqmTQYO8FXflSg9+ixbFuaigAG6+GX79a/jnP+H44z0pioiISMYo9Emd7bsvvPiiB77DDoOlS+NcZAZXXQW33w7PP++jQFauTHtZRURExCn0Sb0ceCA895z37dtpnjv/fF/Y9733/E1z56a1nCIiIuIU+qTeRo3ycRvTp3vwW748wYUnn+ydAZcsgf331+odIiIiGaDQJw3y7W/DE0/4/H2HHJKgjx/AwQd7bV+HDt4mfO+96SymiIhI3lPokwY76ijv4zd/PowcCZ9/nuDC3r1h8mQPgGefDZdfrildRERE0kShT5Ji9GhvwV25Eg46CGbMSHBhSYknxHPPheuvh5NOgvXr01pWERGRfKTQJ0mz//6+GtvWrV6Z958X1YG4AAAgAElEQVT/JLiwsBDuvBNuvBGeesrbhb/6Kq1lFRERyTdZE/rMrJuZ3WNmX5nZJjObZ2Y3m1lJA+55sJltM7NgZr9JZnklvkGDYNIkKC722r9JkxJcaAaXXALPPAMzZ/q6vQlTooiIiDRUVoQ+M+sFTAHOBt4HbgI+Ay4CJptZ+3rcszVwH7AhiUWVWujdG956C7p08VG9L7+8k4uPPhreftsndB4xAv7yF63ZKyIikgJZEfqA24FOwIUhhGNDCJeFEMbg4a8vcF097nkL0Bb4XfKKKbW1224+M0vfvp7rnnhiJxcPGQIffOBzwFxwgY8MWbw4XUUVERHJCxkPfWbWEzgcmAf8pdrpq4H1wBlmVlyHe34HrzW8EFBnsQzp1AkmTIBhw3yqvptv3kkl3i67wAsvwK23wmuveTvxc8+ltbwiIiK5LOOhDxgT2Y8PIWw3f0cIYS3wNtASGF6bm5lZJ+Bu4OkQwgPJLKjUXbt2vlbv0Ud7F77vfW8ng3XN4Kc/hSlTvG346KN9RY8NaqEXERFpqGwIfX0j+9kJzs+J7PvU8n534Z/rvIYUSpKnuBiefBJ++1v4179g+HCYnei/NsDAgfD++3DppT7Kd599PAiKiIhIvWVD6Gsb2a9OcD56vF1NNzKzHwDfAX4cQqhTpzAzO9fMys2sfOnSpXV5q9RCQYHPxfzSS75qx7BhPnA3oebN4Y9/9Kbedes8KV5/PWzblrYyi4iI5JJsCH01sch+p0M6zaw7cDPwWAjh0bp+kxDCXSGEshBCWceOHetcSKmdb33LK+369IFjj4Urrqghx40ZAx9+CMcd56lxxAhN7SIiIlIP2RD6ojV5bROcb1PtukTuATYCP05GoSR19tjDp3T50Y/gd7+DsWNhp5WrpaXeLvzAA77G27BhPsp35cq0lVlERKSxy4bQNyuyT9Rnb8/Ifme9wAD2wad9WRqZjDmYWQD+ETl/ZeTY0w0rriRDixZw113wt795ANx3X+/Gl5CZjwKZNQt+8hO44w6fD+a++zSvn4iISC1kQ+ibENkfbmbblScywfIIvAbv3Rrucz/w9zjbm5HzUyNfv5KcYksy/PCHVXMzDx8OJ54I5eU7eUO7dj6tS3k59OoFZ53la7599FG6iiwiItIoZTz0hRDmAuOB7sBPqp2+FigG7g8hfDPRh5n1M7N+1e5zYQjhnOobVTV9z0eOVZ8LUDJs3329m97ll8Orr3rr7WGH+RiOhJV4e+/tafFvf4MZM/zrSy6BNWvSWnYREZHGIuOhL+LHwBLgVjN72sx+Z2avA5fgzbpXVrt+RmSTHFFaCtddB198AX/4A3zyiQe//fbz1TwqK+O8qaDAqwpnz4ZzzoFbboF+/eDeexO8QUREJH9lReiL1PaVAfcC+wM/A3oBtwIHhBCWZ650kk5t2sD//Z+P1/jrX32sxoknwoABcM89sHlznDeVlvp8fu+9B7vvDmef7dWFb7yR9vKLiIhkq6wIfQAhhAUhhLNDCF1CCM1CCHuEEC4KIayIc62FECzefeJce2/k+quSX2pJlRYt4NxzfdzGv/4FLVt6pV6fPvCPf8DWrXHeNGwYvPMOPPigDwceNQqOPx4+/TTdxRcREck6WRP6ROJp0sTX7Z0yBV58ETp2hB/8APbay8PgDq24BQVw2mmeFn/zG18DbsAAX91j1aqMfAYREZFsoNAnjYKZz+f3/vvw1FNQWAinnOIrtP3733EGfBQVwZVXwpw5cMYZcOON0Ls3/OUvCaoJRUREcptCnzQqZr6Sx9Sp3oq7fj0ccwwccICP9t1Bly7w9797VeGgQT6p8557+pJuS5akvfwiIiKZotAnjVKTJt6KO3063H03fPWVj/YdPRpeeCFOs+/ee8Prr8Ozz0L37j4/zG67+U3eeksTPIuISM5T6JNGrbDQZ2uZPdtnbJk1C448Evr3h9tvh3XrYi42g6OPhgkTPC2ed54nxIMPhsGDvelX8/yJiEiOUuiTnNCiBVx4Icyb582+bdv6am3duvkUMPPnV3tD//6eEr/80id4bt7cm3533RX+5388PYqIiOQQhT7JKc2aeYvte+/57C3f/jbcdBP07Onz/U2aVK0lt7jY54IpL/dRIiefDPff76HwtNN8lmgREZEcoNAnOcnMB3f8618+0fP//Z936TvoID8+fnycbnzDhvkM0PPnw89/7v3/Bg2Ck06CadMy8jlERESSRaFPct5uu/lg3QUL4I47YNEirwE85JAEi3Z06uRvmD/fp30ZPx6GDvVhw1OmpL38IiIiyaDQJ3mjuNjHbsyeDX/+sy/UMWoUfOtb8O67cd7Qvj38+tfeUfCaazwhlpX5SJG4bxAREcleCn2Sd5o390Eec+f6nM3TpnmT71FHwX/+E+cNJSVw9dVe83fddd5h8IADYMQIeOIJ2LYt7Z9BRESkrixofrIdlJWVhfLy8kwXQ9Jk3Tq47Tb44x9h5UpvxT3oIOjc2ed27tLFX7dt630FWbfO+/7dcgt89hn06AEXXeTrw7VunemPIyIiecbMpoQQymq8TqFvRwp9+Wn1ah/pe8st8ZfpbdGiKgDusgu0L62k/YpPKZ36Gu3nTaF90UZKjxlJ+x8eS/u9utC5cyQkioiIpJBCXwMo9OW3EDwALlrk29df7/h68WJYsQKWL4fNm+Pfp1+3tVx4oXHG+a1o1Sq9n0FERPKHQl8DKPRJbYUAGzZ4+Fu+HFbMWMzyR15h0Ssf88+KE5lCGW0L1vDDAe9ywZlr6HHcUOjVS1WAIiKSNAp9DaDQJw1WUUH4oJx3H/qMW57pzuOLDqSSAo7hWS4q+SejRoGNOgS+9z0fJRz/FjRt6puIiEgiCn0NoNAnybbwi0ruvG45f32oFcvWFTGocCbf33I3lU2bs3TgKJb23J8lm9qydCnfbOvXQ5s2cPjhcMQRvnXpUrvvt3GjLzDy1luwdasvUVdamtrPKCIimaHQ1wAKfZIqGzfCI4/4YJHoIh/NqaAjS+nYZhMd9yyhY79SOnY0OnTwKQJffNGXCAbYe28Pf+PGwf77V9UCrlgBb7/tIW/SJF9VbsuWqlbkkhL41a98WWHVHIqI5BaFvgZQ6JNUC8EHhLRpA8Xrl2B33gG33w5LlsDgwXDJJXDqqdC8OSHARx/BCy/49s47PjVgSYlPLTN3btUSwYWFvprcQQfByJE+leCCBX6711+HgQPh5pvhsMMy+/lFRCR5FPoaQKFPMqKiAh56yOeN+fhjnxfmrLN84sD99oMCn0t91Sp45RUPgJMm+biQgw7ybdgwKCra8dYhwDPPwM9+5lMLHnMM3HAD9O6d3o8oIiLJp9DXAAp9klEhwGuveZXcSy95tV6nTr5kyDHHeDVdcXG9bl1R4be97jrYtAkuvhiuusprHGNt3hwZjRyZlmbrVjjwQJ+rUEREdm7pUnjsMXj4YbjgAvjud1P7/RT6GkChT7LGypUe/J591qv21qzx5HXYYR4Ajzqq9qM7YixaBFdcAffe63ly8ODtQ966dTu+p21bOOEEb3UePRqaNGn4x9u82Wsrn3sOnn/eyzVyJBxyiK+LvO++6oMoIo3DmjXw1FMe9F591f+9PmCAL91+0kmp/d4KfQ2g0CdZafNmH6nx7LO+zZvnx/fbzwPgMcfAXnvVaQ7A8nL45S89W7Zv71tpadXr6NcVFf6v1qeegrVrfVWSk0/2ALj//nWbdnDxYh+c8vzz8PLLfr9mzTxI7rGHh8Dp0/3aVq08BI4a5ds++3i/RfCaytWr/Rdt7H7LFr9P794asZxLvvoKHnzQfwaHDoVf/ML/O2dCCD46fskSOPRQaNkyM+VoTDZt8v7Hn3/uXzdt6v8vx9sXFfk/Mtu2rfr/vT62bvXfL2vWxN8qK/13T6KteXNvVGnVylfYbNXKj8fauNF/lz38sO83bYLu3eGUU/z346BB6ZmWVaGvART6JOuF4P3+nnkG/v1v/wsE/tsmGgAPPrhhvzHjiPcLrmdP/wU3bJgHrs2b4+9Xr/Z//X7wgRd/113hyCN9O/RQtlu1ZPFiePNNmDjRt2gILC72P7CrVydeCSVWSYn3eezd27fo6379oEOHpD6aOlu2zD9n//7fdNeUajZs8B/x++7zfqyVlTBkiP88hADf/z5cdln6+qbOnw8PPAD33w+zZ/uxli39Z/ikk3xUfT17XuSEELxZc+ZMmDWraj9rlvclrqys+z1btqwKgO3aVb3ets1/HyXaNmzwLdkKC6sCYKtWPlBu7Vrvgh39h/Dw4emff1+hrwEU+qTRWbTI20iffdaTVUWF/2aMTvC3//6w555JTRerV2/flFHTL3QzL0Y06A0dWvtfjNEQ+NZbHiDbtPGPF2/fpIlXgs6dC59+6tvcuX4stoydO/u/wmO3AQPiD4TZvNmX4Ituixb5H5ZddvEtuh5zaemOn2nzZv/j9+GH22+LFvn53XevqhUYMiS7FmsJAWbM8B+t8nJ/vqWlVVu0Jji6denS8Ob4ykqv7b3/fnj0Uf+DuvvucMYZcOaZ0KcPLFwIf/gD3H23P9/TToMrr/Qwn2xr18ITT3h5JkzwY4cc4mXZfXd48kk/v2SJB5Rx4zwAHnnk9gEwGoiiP5PRbdUqDwmjR3ulffPmyf8MiWzY4FM9vf66P/PNm73M0dqt6OvoVljo/9+vWuXbypU7vq6oqLp/ixb+36tvX9/69fN/eDVp4v8fb90af79xY9X3id1HX69Z4/coKkq8RcNimzZVW+vW279u2tQ/c7xt0ybf1q/37i5r126/j77u0MHD3qhRyenyUl8KfQ2g0CeN2vr1nsKefdZrAZcu9eNt23onuWHDqrbddktKyliyxP8QN2/ufxiaNavax77OZP+8zZu9pubTT72m6KOPfJs+veoPVUGBZ+N+/fyXejTgrVhRu+9RWFgVBDt29PkVZ8zwP2bgz2DgQA93gwf7H54nn/RarK1b/fueeqpve+5Z/8+6dq1/3xkz/POtW+ffNxpu27VL/N5NmzxgP/ec//hEm+N69PDntLP1ptu29e6mRxwBY8dC1661K++SJR463nrLa/Y+/9xDxkknebg65JD4/175+mv405/gjjs8KJx0kg9MGjSo5u8ZQtUf94oK32JfL14M//qX//fZsMFrE888E04/3Z9FrG3bvOyPPeYBcPFiDx5jx/rPfDTgrV1b9Z6CAm+eLi72KZdC8JB04IEeIEaP9v9FkxkCKypg8mQPrxMmwHvvechq2hTKyjwMrV8ff4v+DDdp4jXo7dptv5WU+H//bt3857hvXw/FqsVOD4W+BlDok5yxbZv/1f/gg6rtww/9Nz34KI5hw/wv9Xe+s+NfszywbZv/Qf7ww6ogOHu2/wHs0sVr8eLtmzf3sLJ4sYeP2H1069zZw93gwR709twzfov7smUeFh5+2ANXCN5/8dRT/T9PNJfH5vPo6y1bqoJsNOQtWFB1XbNmHibWrKk61q3b9jWc/fv75/73v2H8eA+J0fFCRx3ltVbduvl7Q/CAtWLF9tuyZd7L4KWXqiYTHzSoqrL5wAO9LCF4+H7rLd/efNOb/8C/58EHe63eccfVvql06VKf6ejPf/ZgNWqU36t6c1/s17E1Uom0a+ejLr///do32W3b5gH2scf8eTZvXtW9IHbr3r2qf9iKFf4sot0Zpk3z51RU5M+tVy8PT/E2M99XVvr3ju5jX1dWek33O+94sC0o8H//jR7t28iR23eviCfaTaNly+yqjRan0NcACn2S0yoqPOFEQ+B773n7I3gyOfZY37KtrTFPLFzozZoPP+xNqrVVVOThbcCAqv2AAd7nskkTv2801Ea3GTOq8j94zdxRR/k2Zkz9BihEu5u++KJvkyZ5LVHr1t68P3OmlwW8ZmjECA96Bx3kQaQhNVsrVsCtt8LTT3ugqt7cV70JsEUL/36x++jrVq086GVimqIVK7bv07p4sQe36lsIVa+jIbBJE9+qv+7Y0WtMR4/25922bfo/l6SOQl8DKPRJ3pk719vVnn7a/0qH4G1P0QA4cqTmTsmAuXO9Vgz8P0lU7OuCAq+g3WOPujelbdnitZozZnhtUl36WdbW2rU+7eSLL3rTYr9+HvAOPtgHm2eyH5RIrlDoawCFPslrS5Z4h66nn/a2vk2bvJ2tc+fte+1X79E/YIC3Sap2UEQkrRT6GkChTyRi3ToPfm+84Z22YjtxLV/uQ+lif4fsvjscf7xvBx6oahwRkTRQ6GsAhT6RWtq2zedRWLbMe4k/+WRV7WCnTt40fPzx3pGo+qymIiKSFAp9DaDQJ9IAa9d6B64nn/QZnNet817jRx3lHbnKynxYZ5InjhYRyVcKfQ2g0CeSJBUVPgndk0/6/BXLl/vx5s19dHBZmc9JUlbmQ07VHCwiUmcKfQ2g0CeSAiH4rLsffOBzkZSXw5QpVTPWtmzpQTA6fX9069UrvcsUiIg0Mgp9DaDQJ5ImlZU+Z0g0BE6b5jP1Rtcog6o5SaIhcOhQX6+qTx9N9y8iQu1DnybeEpHMKSjwidv69fP1raLWrPEwGF2tPbpNmODLKYAvmTFsmAfA6Lbrrpn5HCIijYBCn4hknzZtvJ9fWbV/uG7b5uHv/fertj/+sWph0F139fcMGFC1wnvfvr4wqIhInlPzbhxq3hVpRCoqYOrUqhA4ZYovRhsNguDTx0Sbh/v18+UrOnb0rVMnn2BaTcUi0kipeVdE8kOLFr5I6vDhVce2bPFBI7Nm+WKv0ebhp5/2OQWrKyjwVUWiQbBzZ1+o9OijoVu39H0WEZEUUk1fHKrpE8lhy5fDwoWwdGnibf58+OILv36ffTz8HXMM7L23lpkTkayj0bsNoNAnkudC8JrBZ5/17Z13/FjXrlUBcPRor2UUEckwhb4GUOgTke0sXeqri/z73/Dyy7B+vc8duM8+3qx8wAG+79ZNNYEiknYKfQ2g0CciCVVU+NQxr78Okyf7wJGKCj+3667bh8B+/byvoIKgiKSQBnKIiKRCixZwxBG+AWzeDB9+CO++69vkyb7sXFTLlj5aePfdd9zvuis0awZNm1ZtTZps/3VBgUKjiCSFavriUE2fiDTIkiU+fcynn/qAkOjAkPnzvam4Lnr1glNOgdNO8/kHRUSqUfNuAyj0iUjKbNgACxZ4AFy0yOcT3LbN99Vfb9kCb78Nr77qS9YNGQKnnuohcI89Mv1JRCRLKPQ1gEKfiGSVxYvh0UfhoYe8CRlg5EgPgCed5HMLikjeUuhrAIU+Eclan30GjzwCDz4I06f7sWbNfDRxs2bbv47uS0qqViOJbl27qq+gSI5Q6GsAhT4RyXohwEcfwXPPwZo1PqBk06bt99HXy5b5yiRr1lS9v7h4+yA4cCAMGgQ9e/pgEhFpNDR6V0Qkl5nB4MG+1UYI3kw8c+b229tve7NxVFFRVQCM3XbZJTWfQ0TSJmtq+sysG/ArYCzQHlgEPA1cG0JYWYv3FwPHAkcC+wC7AZXALOBh4LYQwubalEU1fSKSV9avhxkzvOYwdlu8uOqatm2hTRuvISwu9qloqr8uKfFw2Lmz76Ov27VTU7JICjWq5l0z6wW8A3QCngFmAvsBo/HQNiKEsLyGe4wFXgRWABOAT4FS4Gigc+T+h4YQKmoqj0KfiAg+vUw0AM6Z4+Fw/XofgRzv9cqVPuq4umbNPAB26eKrmBxwgG+9eysMiiRBYwt9LwOHAxeGEG6LOX4jcAnw1xDCeTXcYygwEHgstkbPzFoDE/Hav0tDCDfUVB6FPhGReqis9OD39ddeS7h4cdXrr7/2qWrKy6v6FnboULWCyQEHwLBh0KpVZj+DSCPUaEKfmfUE5gLzgF4hhMqYc63xZl4DOoUQ1tfze5wGPAg8F0I4uqbrFfpERFKkstJHHU+eXLXNnOnnmjTxCaj32sv7FUY3DS4R2anGNJBjTGQ/PjbwAYQQ1prZ23gt4HDgtXp+jy2RfZx2BxERSZuCAg91e+0FP/qRH1uxomoJu//8B955Bx5+uOo9LVr4COMBAzwE9u7tk1PvsYc3G6uJWKRWsiH09Y3sZyc4PwcPfX2of+j7QWT/Uj3fLyIiqVJaCuPG+Ra1dq0PLvnkk6rtrbe2H2kMPhdhNABGt1139SAYQuKtZcvtB5106OBrHYvksGz4CW8b2a9OcD56vF19bm5mF+AjgqcC9+zkunOBcwF23333+nwrERFJltatYb/9fIu1di3Mm+fb/PlV27x5MG2ar3tcH2Ye/KKjjjt08EEpGzf6YJXoFvt169Zw0EEwahQccgj06aNaR8lq2RD6ahL9P6jOnQ/N7HjgZuBr4IQQwpZE14YQ7gLuAu/TV49yiohIqrVuXTV3YDwbN1ZNNWO241ZQ4OfWrdt+wEn1bf58H3XcsqVvJSW+iknLlj6XYcuWHjAnTqxqiu7cGQ4+2APgqFHQv3/8ELh1K1RUVG1btvhE2tF97OstW6BbN2/WVqCUBsqG0BetyWub4HybatfVipkdCzwCLAFGhxA+q1/xRESk0Sgqgu7da3ftnns2/PuF4NPZvPFG1fboo36uQwef3zA24FVUwLZtdf8+nTp5kBwzBkaP9rLXNgRGB2wqNOa9bAh9syL7PgnOR/+vTNTnbwdmdhLwEF7DNyaEMKf+xRMREUnAzJt1+/TxgSkhwOefew3g22/7MngtWiTeomskFxZWrZ0c+7ppU5g1C15/3bdooOza1cPfmDE+3c369fDll7BwYfw9wN57Q1lZ1da7d1XNp+SFbJiypRc+kfI8Ek/ZUgB0rM2ULZHpWe4HvqSeNXyaskVERLJOCPDppzBhggfACRPi92Fs2tQnwu7WzcNht27eTDxlCkyd6rWN4Cus7LNPVQjcd1/o1Us1go1Qo5myJYQw18zG4yN0fwLcFnP6WqAYn5z5m8BnZv0i750Zey8z+z4+WGM+Hvjmp7j4IiIi6WHmzbp77gnnnushcPp0+OADX+ouGvI6dUo8r+GWLT4qurzcQ2B5Odx2m9dIgjdH77OPB8B99/UwqCCYMzJe0wdxl2GbAeyPL8M2Gzgwdhk2MwsAIQSLOTYaeBWvFbwHWBDnW60KIdxcU3lU0yciInljyxb4+GMPgdFt2jQfSAJVQXDAAG96jjdAJrq1aePBc9ddqzatvZxyjWZFjigz2w34FT69Snu8Wfdp4NoQwopq18YLfWcB/6jh28wPIXSvqSwKfSIiktc2b/a5EaO1gVOm+ICVysqdz3+4efOO9yoqqgqAXbtCjx7en7BXL9936VK3voWVlT4Cetu2qn3s661b/X4dOni/yTzQ6EJfNlHoExERqYcNG2DRIvjqq6rtyy+rXi9c6NPhbI1ZIKtFC19qLxoES0pg1SrfVq7c8fXq1VUjkmvSpo3Pu9ip045bcbGHw+hUPtX3hYVepj59fLBNFms0ffpEREQkR7Rs6cGtV6/E12zdCl98AXPn+sCU2P0rr/hci9G5Edu1861rV5+rsKTEm5tbtPB+i02bxt9v2wbLlvmci0uW+DZnDkya5MfrUuFVUODhr39/b+KObv36eTkbEYU+ERERSZ+mTb1mr2dP+Na3tj8XgvcxbNYsdd8/Ggg3bvTvF22yrr6vqPCgOH161fb881W1lGbQvr3XCDZtWrWv/vqCC+D001P3eepAoU9ERESyg1lqAx94TeAuu9Tu2n322f7rLVu8VnL6dO/zuGiRh8DYbcuW7V+n+vPUgUKfiIiISG0UFnozb//+cMIJmS5NnWkqbhEREZE8oNAnIiIikgcU+kRERETygEKfiIiISB5Q6BMRERHJAwp9IiIiInlAoU9EREQkDyj0iYiIiOQBhT4RERGRPKDQJyIiIpIHFPpERERE8oBCn4iIiEgeUOgTERERyQMWQsh0GbKOmS0F5qf423QAlqX4e+QjPdfU0HNNDT3X1NBzTQ0919RIxnPdI4TQsaaLFPoyxMzKQwhlmS5HrtFzTQ0919TQc00NPdfU0HNNjXQ+VzXvioiIiOQBhT4RERGRPKDQlzl3ZboAOUrPNTX0XFNDzzU19FxTQ881NdL2XNWnT0RERCQPqKZPREREJA8o9ImIiIjkAYW+NDKzbmZ2j5l9ZWabzGyemd1sZiWZLlu2M7MTzew2M3vLzNaYWTCzB2p4z4Fm9oKZrTCzDWb2oZldbGZN0lXubGZm7c3sHDN7ysw+NbONZrbazCaZ2Q/NLO7vBz3XmpnZ783sNTNbEHmuK8zsv2Z2tZm1T/AePdd6MLMzIr8Pgpmdk+Cao8xsYuTne52ZvWdm3093WbNV5G9RSLB9neA9+nmtJTM7yMyeMLNFkb/9i8xsvJmNi3NtSp+r+vSliZn1At4BOgHPADOB/YDRwCxgRAhheeZKmN3MbCowBFgHLAT6AQ+GEE5PcP13gCeACuBfwArgaKAv8HgI4aR0lDubmdl5wB3AImAC8AWwC3A80BZ/fieFmF8Seq61Y2abgf8A04ElQDEwHCgDvgKGhxAWxFyv51oPZrYb8BHQBGgF/CiE8Ldq11wA3AYsx5/tZuBEoBtwQwjh0rQWOguZ2TygHXBznNPrQgh/qna9fl5rycyuAn6NT778HP77tgOwNzAhhPDzmGtT/1xDCNrSsAEvAwH4abXjN0aO35npMmbzhofjPQEDRkWe2QMJrm2D/6HdBJTFHG+BB+8AnJLpz5TpDRgT+YVSUO14ZzwABuAEPdd6PdsWCY5fF3lOt+u5NvgZG/AqMBf4Y+Q5nVPtmu74H9DlQPeY4yXAp5H3HJDpz5LpDZgHzKvltfp5rf1zPSnyPF4BWsc5X5ju56rm3TQws57A4fj/WH+pdvpqYD1whpkVp7lojUYIYUIIYU6I/F9QgxOBjsAjIYTymHtUAFdFvjw/BcVsVEIIr4cQ/h1CqKx2/GvgzsiXo2JO6bnWUuSZxM92SUQAAAl0SURBVPNoZL9nzDE91/q5EP+Hy9n479B4fgA0B/4cQpgXPRhCWAn8NvLleSksYy7Sz2stRLrH/B7YAJwWQlhb/ZoQwpaYL9PyXJs29AZSK2Mi+/Fx/sCuNbO38VA4HHgt3YXLQdHn/VKcc2/i/xMeaGbNQwib0lesRiX6y2hrzDE914Y7OrL/MOaYnmsdmVl/4HrglhDCm2Y2JsGlO3u2L1a7Jt81N7PTgd3xEP0h8GYIYVu16/TzWjsHAj2Ax4GVZnYksBde8/x+CGFytevT8lwV+tKjb2Q/O8H5OXjo64NCXzIkfN4hhK1m9jkwEOgJzEhnwRoDM2sKnBn5MvYXkJ5rHZnZpXhfs7Z4f76R+B/T62Mu03Otg8jP5z/xLghX1HD5zp7tIjNbD3Qzs5YhhA3JLWmj0xl/rrE+N7OzQwhvxBzTz2vtDIvsF+P9ewfFnjSzN4ETQwhLI4fS8lzVvJsebSP71QnOR4+3S0NZ8oGed8Ncj/+L9IUQwssxx/Vc6+5SvAvHxXjgewk4POYXPei51tUv8U7wZ4UQNtZwbW2fbdsE5/PFP4BD8eBXjAeUv+J9Il80syEx1+rntXY6RfbnAUXAYUBr/Hfry8DBwGMx16fluSr0ZQeL7DWUOj30vBMwswuBn+Gjy8+o69sjez3XiBBC5xCC4X9Mj8f/lf5fM9unDrfRc40ws/3w2r0b4jSP1euWkX1eP9sQwrWRPr6LQwgbQggfhxDOwwcaFgHX1OF2eqYuOsWK4TV6r4UQ1oUQPgGOw2ehOMTMDqjl/ZLyXBX60qOmf022qXadNIyedz2Y2U+AW/BpRkaHEFZUu0TPtZ4if0yfwrtxtAfujzmt51oLMc26s4H/V8u31fbZrmlA0XJZdEDXwTHH9PNaOysj+89CCNNiT0RqqKOtKPtF9ml5rgp96TErsu+T4Hx0JF+iPn9SNwmfd+QPRw98gMJn6SxUNjOzi4E/Ax/jgS/ehKx6rg0UQpiPh+qBZtYhcljPtXZa4c+oP1ARO4Ew3oQOcHfkWHS+uZ092y54U+ZC9edLaElkHzuzhH5eayf6nFYlOB8NhUXVrk/pc1XoS48Jkf3h1Vc5MLPWwAhgI/BuuguWo16P7MfGOXcw0BJ4J89Hln3DzH4B3ARMxQPfkgSX6rkmx66RfXRUpJ5r7WwC/p5g+2/kmkmRr6NNvzt7tkdUu0Z2FG16jA0a+nmtnTfxkLanmTWLc36vyH5eZJ+e55rpyQvzZUOTMyfzWY6i5smZl6LJQ2vzLP9f5HmUA6U1XKvnWrtn2g/oHOd4AVWTM7+t55rUZ34N8Sdn7oEmZ67p2Q2M9/8+sAc+s0QArog5rp/X2j/bByLP4zfVjn8LqMRrAdul87lqGbY0ibMM2wxgf3ylidnAgUHLsCVkZscCx0a+7Ax8G//X51uRY8tCzHJKkesfx3/hP4IvZ3MMkeVsgJNDnv/wR9YevRevcbqN+H1F5oUQ7o15j55rDSJN5X/E/6U/Fw8cuwCH4AM5vgYODSFMj3mPnmsDmNk1eBNvvGXYfgrcipZhiyvy7C7DW6Q+B9YCvYAj8cDxAnBcCGFzzHv081oLZtYJeBvojf+teh8P08fhIe60EMJjMden/rlmOgnn0wbshg+NX4T/4pmPd5zfaQ2Ltu3+Jf//27uTUDmKOI7j3x+RqAdNFMQNwYPBBVEjhIgGjSgSD3EBJYiaIHoQTwoKIoJPRMhFchMPIiJqTC7uKChi1CAuaFBwiYKRkKiJW9wT1L+H7oFhmMnL6DNj7O8HmmKqq2uq682b93/VXdWjtk1DjjmL5gvrO5rL5+8DNwGzJn0+/4VtD/q0gJft17H79WSaJ+9soHne5u80AfVbbZ8P/X23X/9Rn/c+y9eN2L8UWEcT0Pzc/ixWTLrd/4WN5p+R1TQz9r+nWZh9O82jw5ZDMzg05Dg/r3vWv4fSXNH7rP27/w3NwM8Zk+hXR/okSZI6wIkckiRJHWDQJ0mS1AEGfZIkSR1g0CdJktQBBn2SJEkdYNAnSZLUAQZ9kiRJHWDQJ0n7oCRTSSrJ4km3RdK+waBPUie1AdN02+JJt1OSZsp+k26AJE3YnbvZt2lvNUKS/m0GfZI6raqmJt0GSdobvLwrSXug/x66JCuSvJvk1yTbkjyQ5IgRx81L8lCSLUl2Jdnavp43ovysJNcnWZ9kR/senya5fzfHXJbkzSS/JPk2yWNJjp7J85e073OkT5LGcxNwAbAGeB5YBFwDLE6ysKq29womWQC8CBwEPAV8AJwAXAlcnOS8qnq7r/xs4FngfGAz8CjwA3AscCnwGvDJQHtuAC5q618HLASWAacmOa2qds7kyUvadxn0Seq0JFMjdv1WVSuH5F8ILKyqd/vqWAXcCKwErm3zAjwEHAxcVVWP9JVfBjwGPJzkpKr6s901RRPwPQ1c3h+wJdm/rWvQEmBBVb3fV/ZR4ArgYmDtyJOX1Cmpqkm3QZL2uiTTffntqKq5feWngDuAB6rq2oG65gCfA/sDc6tqZ5KzaEbmXq+qM4e8/6s0o4TnVNUrSWYB3wCzgeOqaus07e+15+6qun1g37nAS8A9VXXzNOcpqSO8p09Sp1VVRmxzRxyybkgdO4ANwAHAiW326W360oh6evnz2/QEYA7w3nQB34C3h+RtbtNDxqhH0v+cQZ8kjeerEflftumcgfSLEeV7+XMH0i1jtuf7IXm/t+msMeuS9D9m0CdJ4zl8RH5v9u6OgXTorF7gyIFyveDNWbeS/hUGfZI0nnMGM9p7+k4DfgM+bLN7Ez0Wj6inl/9Om35EE/idkuSomWioJPUz6JOk8VydZP5A3hTN5dzVfTNu1wMfA4uSXNZfuH19NrCRZrIHVfUHcC9wIHBfO1u3/5jZSQ6b4XOR1CEu2SKp03azZAvAE1W1YSDvOWB9krU09+UtardNwK29QlVVSVYALwBrkjxJM5p3PHAJ8COwvG+5FmgeCbcQWApsTPJMW+4YmrUBbwEe/FsnKqnzDPokdd0du9m3iWZWbr9VwOM06/ItA36iCcRuq6pt/QWr6o12gebbadbfWwp8DawG7qqqjwfK70qyBLgeWA6sAAJsbd/ztfFPT5IartMnSXugb128c6vq5cm2RpLG5z19kiRJHWDQJ0mS1AEGfZIkSR3gPX2SJEkd4EifJElSBxj0SZIkdYBBnyRJUgcY9EmSJHWAQZ8kSVIHGPRJkiR1wF83HYSpGRxZgAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(1, 1, figsize=(10,6))\n",
    "ax.plot(np.sqrt(model_history.history['loss']), 'r', label='train')\n",
    "ax.plot(np.sqrt(model_history.history['val_loss']), 'b' ,label='val')\n",
    "ax.set_xlabel(r'Epoch', fontsize=20)\n",
    "ax.set_ylabel(r'Loss', fontsize=20)\n",
    "ax.legend()\n",
    "ax.tick_params(labelsize=20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 5 - Random restarts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This method does not seem to have an implementation in `keras`. Develop your own function for this using `keras.callbacks.LearningRateScheduler`. You can refer back to how we used it to set a custom learning rate.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tuning the Hyperparameters using Cross Validation\n",
    "\n",
    "Now instead of trying different values by hand, we will use GridSearchCV from Scikit-Learn to try out several values for our hyperparameters and compare the results.\n",
    "\n",
    "To do cross-validation with `keras` we will use the wrappers for the Scikit-Learn API. They provide a way to use Sequential Keras models (single-input only) as part of your Scikit-Learn workflow.\n",
    "\n",
    "There are two wrappers available:\n",
    "\n",
    "`keras.wrappers.scikit_learn.KerasClassifier(build_fn=None, **sk_params)`, which implements the Scikit-Learn classifier interface,\n",
    "\n",
    "`keras.wrappers.scikit_learn.KerasRegressor(build_fn=None, **sk_params)`, which implements the Scikit-Learn regressor interface."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from keras.wrappers.scikit_learn import KerasClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Trying different weight initializations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's create a function that creates the model (required for KerasClassifier) \n",
    "# while accepting the hyperparameters we want to tune \n",
    "# we also pass some default values such as optimizer='rmsprop'\n",
    "def create_model(init_mode='uniform'):\n",
    "    # define model\n",
    "    model = Sequential()\n",
    "    model.add(Dense(64, kernel_initializer=init_mode, activation=tf.nn.relu, input_dim=784)) \n",
    "    model.add(Dropout(0.1))\n",
    "    model.add(Dense(64, kernel_initializer=init_mode, activation=tf.nn.relu))\n",
    "    model.add(Dense(10, kernel_initializer=init_mode, activation=tf.nn.softmax))\n",
    "    # compile model\n",
    "    model.compile(loss='categorical_crossentropy',\n",
    "              optimizer=RMSprop(),\n",
    "              metrics=['accuracy'])\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "60000/60000 [==============================] - 1s 21us/step - loss: 0.4118 - acc: 0.8824\n",
      "Epoch 2/10\n",
      "60000/60000 [==============================] - 1s 15us/step - loss: 0.1936 - acc: 0.9437\n",
      "Epoch 3/10\n",
      "60000/60000 [==============================] - 1s 14us/step - loss: 0.1482 - acc: 0.9553\n",
      "Epoch 4/10\n",
      "60000/60000 [==============================] - 1s 14us/step - loss: 0.1225 - acc: 0.9631\n",
      "Epoch 5/10\n",
      "60000/60000 [==============================] - 1s 14us/step - loss: 0.1064 - acc: 0.9676\n",
      "Epoch 6/10\n",
      "60000/60000 [==============================] - 1s 14us/step - loss: 0.0944 - acc: 0.9710\n",
      "Epoch 7/10\n",
      "60000/60000 [==============================] - 1s 14us/step - loss: 0.0876 - acc: 0.9732\n",
      "Epoch 8/10\n",
      "60000/60000 [==============================] - 1s 15us/step - loss: 0.0809 - acc: 0.9745\n",
      "Epoch 9/10\n",
      "60000/60000 [==============================] - 1s 14us/step - loss: 0.0741 - acc: 0.9775\n",
      "Epoch 10/10\n",
      "60000/60000 [==============================] - 1s 15us/step - loss: 0.0709 - acc: 0.9783\n",
      "CPU times: user 21 s, sys: 3.56 s, total: 24.5 s\n",
      "Wall time: 1min 20s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "seed = 7\n",
    "numpy.random.seed(seed)\n",
    "batch_size = 128\n",
    "epochs = 10\n",
    "\n",
    "model_CV = KerasClassifier(build_fn=create_model, epochs=epochs, \n",
    "                           batch_size=batch_size, verbose=1)\n",
    "# define the grid search parameters\n",
    "init_mode = ['uniform', 'lecun_uniform', 'normal', 'zero', \n",
    "             'glorot_normal', 'glorot_uniform', 'he_normal', 'he_uniform']\n",
    "\n",
    "param_grid = dict(init_mode=init_mode)\n",
    "grid = GridSearchCV(estimator=model_CV, param_grid=param_grid, n_jobs=-1, cv=3)\n",
    "grid_result = grid.fit(x_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best Accuracy for 0.9689333333333333 using {'init_mode': 'lecun_uniform'}\n",
      " mean=0.9647, std=0.001438 using {'init_mode': 'uniform'}\n",
      " mean=0.9689, std=0.001044 using {'init_mode': 'lecun_uniform'}\n",
      " mean=0.9651, std=0.001515 using {'init_mode': 'normal'}\n",
      " mean=0.1124, std=0.002416 using {'init_mode': 'zero'}\n",
      " mean=0.9657, std=0.0005104 using {'init_mode': 'glorot_normal'}\n",
      " mean=0.9687, std=0.0008436 using {'init_mode': 'glorot_uniform'}\n",
      " mean=0.9681, std=0.002145 using {'init_mode': 'he_normal'}\n",
      " mean=0.9685, std=0.001952 using {'init_mode': 'he_uniform'}\n"
     ]
    }
   ],
   "source": [
    "# print results\n",
    "print(f'Best Accuracy for {grid_result.best_score_} using {grid_result.best_params_}')\n",
    "means = grid_result.cv_results_['mean_test_score']\n",
    "stds = grid_result.cv_results_['std_test_score']\n",
    "params = grid_result.cv_results_['params']\n",
    "for mean, stdev, param in zip(means, stds, params):\n",
    "    print(f' mean={mean:.4}, std={stdev:.4} using {param}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save Your Neural Network Model to JSON"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Hierarchical Data Format (HDF5) is a data storage format for storing large arrays of data including values for the weights in a neural network.\n",
    "You can install HDF5 Python module: pip install h5py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Keras gives you the ability to describe and save any model using the JSON format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model saved\n",
      "Model loaded\n"
     ]
    }
   ],
   "source": [
    "from keras.models import model_from_json\n",
    "\n",
    "# serialize model to JSON\n",
    "model_json = model.to_json()\n",
    "\n",
    "with open(\"model.json\", \"w\") as json_file:\n",
    "    json_file.write(model_json)\n",
    "\n",
    "# save weights to HDF5\n",
    "model.save_weights(\"model.h5\")\n",
    "print(\"Model saved\")\n",
    "\n",
    "# when you want to retrieve the model: load json and create model\n",
    "json_file = open('model.json', 'r')\n",
    "saved_model = json_file.read()\n",
    "# close the file as good practice\n",
    "json_file.close()\n",
    "model_from_json = model_from_json(saved_model)\n",
    "# load weights into new model\n",
    "model_from_json.load_weights(\"model.h5\")\n",
    "print(\"Model loaded\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cross-validation with more than one hyperparameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can do cross-validation with more than one parameters simultaneously, effectively trying out combinations of them.  \n",
    "\n",
    "**Note: Cross-validation in neural networks is computationally expensive**. Think before you experiment! Multiply the number of features you are validating on to see how many combinations there are. Each combination is evaluated using the cv-fold cross-validation (cv is a parameter we choose). \n",
    "\n",
    "For example, we can choose to search for different values of:\n",
    "\n",
    "- batch size, \n",
    "- number of epochs and \n",
    "- initialization mode. \n",
    "\n",
    "The choices are specified into a dictionary and passed to GridSearchCV. \n",
    "\n",
    "We will perform a GridSearch for `batch size`, `number of epochs` and `initializer` combined."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "# repeat some of the initial values here so we make sure they were not changed\n",
    "input_dim = x_train.shape[1]\n",
    "num_classes = 10\n",
    "\n",
    "# let's create a function that creates the model (required for KerasClassifier) \n",
    "# while accepting the hyperparameters we want to tune \n",
    "# we also pass some default values such as optimizer='rmsprop'\n",
    "def create_model_2(optimizer='rmsprop', init='glorot_uniform'):\n",
    "    model = Sequential()\n",
    "    model.add(Dense(64, input_dim=input_dim, kernel_initializer=init, activation='relu'))\n",
    "    model.add(Dropout(0.1))\n",
    "    model.add(Dense(64, kernel_initializer=init, activation=tf.nn.relu))\n",
    "    model.add(Dense(num_classes, kernel_initializer=init, activation=tf.nn.softmax))\n",
    "\n",
    "    # compile model\n",
    "    model.compile(loss='categorical_crossentropy', \n",
    "                  optimizer=optimizer, \n",
    "                  metrics=['accuracy'])\n",
    "\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "40000/40000 [==============================] - 1s 21us/step - loss: 0.4801 - acc: 0.8601\n",
      "Epoch 2/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.2309 - acc: 0.9310\n",
      "Epoch 3/10\n",
      "40000/40000 [==============================] - 1s 14us/step - loss: 0.1744 - acc: 0.9479\n",
      "Epoch 4/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1422 - acc: 0.9575\n",
      "Epoch 5/10\n",
      "40000/40000 [==============================] - 1s 14us/step - loss: 0.1214 - acc: 0.9625\n",
      "Epoch 6/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1081 - acc: 0.9675\n",
      "Epoch 7/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0974 - acc: 0.9693\n",
      "Epoch 8/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0874 - acc: 0.9730\n",
      "Epoch 9/10\n",
      "40000/40000 [==============================] - 1s 14us/step - loss: 0.0800 - acc: 0.9750\n",
      "Epoch 10/10\n",
      "40000/40000 [==============================] - 1s 14us/step - loss: 0.0750 - acc: 0.9765\n",
      "20000/20000 [==============================] - 0s 10us/step\n",
      "40000/40000 [==============================] - 0s 6us/step\n",
      "Epoch 1/10\n",
      "40000/40000 [==============================] - 1s 22us/step - loss: 0.4746 - acc: 0.8656\n",
      "Epoch 2/10\n",
      "40000/40000 [==============================] - 1s 14us/step - loss: 0.2264 - acc: 0.9336\n",
      "Epoch 3/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1734 - acc: 0.9487\n",
      "Epoch 4/10\n",
      "40000/40000 [==============================] - 1s 14us/step - loss: 0.1436 - acc: 0.9568\n",
      "Epoch 5/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1256 - acc: 0.9614\n",
      "Epoch 6/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1104 - acc: 0.9660\n",
      "Epoch 7/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0973 - acc: 0.9707\n",
      "Epoch 8/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0870 - acc: 0.9733\n",
      "Epoch 9/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0818 - acc: 0.9748\n",
      "Epoch 10/10\n",
      "40000/40000 [==============================] - 1s 14us/step - loss: 0.0730 - acc: 0.9770\n",
      "20000/20000 [==============================] - 0s 10us/step\n",
      "40000/40000 [==============================] - 0s 6us/step\n",
      "Epoch 1/10\n",
      "40000/40000 [==============================] - 1s 23us/step - loss: 0.4639 - acc: 0.8671\n",
      "Epoch 2/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.2208 - acc: 0.9344\n",
      "Epoch 3/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1693 - acc: 0.9491\n",
      "Epoch 4/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1388 - acc: 0.9580\n",
      "Epoch 5/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1206 - acc: 0.9634\n",
      "Epoch 6/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1062 - acc: 0.9678\n",
      "Epoch 7/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0956 - acc: 0.9711\n",
      "Epoch 8/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0870 - acc: 0.9728\n",
      "Epoch 9/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0794 - acc: 0.9750\n",
      "Epoch 10/10\n",
      "40000/40000 [==============================] - 1s 14us/step - loss: 0.0748 - acc: 0.9774\n",
      "20000/20000 [==============================] - 0s 11us/step\n",
      "40000/40000 [==============================] - 0s 7us/step\n",
      "Epoch 1/10\n",
      "40000/40000 [==============================] - 1s 23us/step - loss: 0.7144 - acc: 0.7894\n",
      "Epoch 2/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.3246 - acc: 0.9045\n",
      "Epoch 3/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.2482 - acc: 0.9268\n",
      "Epoch 4/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.2005 - acc: 0.9407\n",
      "Epoch 5/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1673 - acc: 0.9485\n",
      "Epoch 6/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1462 - acc: 0.9559\n",
      "Epoch 7/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1305 - acc: 0.9604\n",
      "Epoch 8/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1166 - acc: 0.9643\n",
      "Epoch 9/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1079 - acc: 0.9675\n",
      "Epoch 10/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0983 - acc: 0.9695\n",
      "20000/20000 [==============================] - 0s 12us/step\n",
      "40000/40000 [==============================] - 0s 6us/step\n",
      "Epoch 1/10\n",
      "40000/40000 [==============================] - 1s 24us/step - loss: 0.6894 - acc: 0.7944\n",
      "Epoch 2/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.3171 - acc: 0.9061\n",
      "Epoch 3/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.2358 - acc: 0.9312\n",
      "Epoch 4/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1911 - acc: 0.9422\n",
      "Epoch 5/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1595 - acc: 0.9526\n",
      "Epoch 6/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1401 - acc: 0.9579\n",
      "Epoch 7/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1230 - acc: 0.9636\n",
      "Epoch 8/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1096 - acc: 0.9672\n",
      "Epoch 9/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1027 - acc: 0.9692\n",
      "Epoch 10/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0936 - acc: 0.9718\n",
      "20000/20000 [==============================] - 0s 12us/step\n",
      "40000/40000 [==============================] - 0s 7us/step\n",
      "Epoch 1/10\n",
      "40000/40000 [==============================] - 1s 24us/step - loss: 0.7028 - acc: 0.7976\n",
      "Epoch 2/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.3189 - acc: 0.9055\n",
      "Epoch 3/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.2390 - acc: 0.9307\n",
      "Epoch 4/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1910 - acc: 0.9435\n",
      "Epoch 5/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1595 - acc: 0.9528\n",
      "Epoch 6/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1376 - acc: 0.9583\n",
      "Epoch 7/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1213 - acc: 0.9629\n",
      "Epoch 8/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1100 - acc: 0.9664\n",
      "Epoch 9/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0987 - acc: 0.9697\n",
      "Epoch 10/10\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0932 - acc: 0.9714\n",
      "20000/20000 [==============================] - 0s 13us/step\n",
      "40000/40000 [==============================] - 0s 7us/step\n",
      "Epoch 1/20\n",
      "40000/40000 [==============================] - 1s 25us/step - loss: 0.4938 - acc: 0.8589\n",
      "Epoch 2/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.2286 - acc: 0.9324\n",
      "Epoch 3/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1751 - acc: 0.9470\n",
      "Epoch 4/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1458 - acc: 0.9553\n",
      "Epoch 5/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1237 - acc: 0.9620\n",
      "Epoch 6/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1106 - acc: 0.9669\n",
      "Epoch 7/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0998 - acc: 0.9696\n",
      "Epoch 8/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0898 - acc: 0.9729\n",
      "Epoch 9/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0799 - acc: 0.9749\n",
      "Epoch 10/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0746 - acc: 0.9769\n",
      "Epoch 11/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0688 - acc: 0.9783\n",
      "Epoch 12/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0655 - acc: 0.9792\n",
      "Epoch 13/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0619 - acc: 0.9801\n",
      "Epoch 14/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0574 - acc: 0.9814\n",
      "Epoch 15/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0549 - acc: 0.9836\n",
      "Epoch 16/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0512 - acc: 0.9837\n",
      "Epoch 17/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0464 - acc: 0.9855\n",
      "Epoch 18/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0466 - acc: 0.9850\n",
      "Epoch 19/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0431 - acc: 0.9863\n",
      "Epoch 20/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0430 - acc: 0.9861\n",
      "20000/20000 [==============================] - 0s 13us/step\n",
      "40000/40000 [==============================] - 0s 7us/step\n",
      "Epoch 1/20\n",
      "40000/40000 [==============================] - 1s 25us/step - loss: 0.4956 - acc: 0.8584\n",
      "Epoch 2/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.2286 - acc: 0.9321\n",
      "Epoch 3/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1746 - acc: 0.9474\n",
      "Epoch 4/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1438 - acc: 0.9574\n",
      "Epoch 5/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1232 - acc: 0.9634\n",
      "Epoch 6/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1104 - acc: 0.9665\n",
      "Epoch 7/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0986 - acc: 0.9696\n",
      "Epoch 8/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0886 - acc: 0.9723\n",
      "Epoch 9/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0828 - acc: 0.9741\n",
      "Epoch 10/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0763 - acc: 0.9768\n",
      "Epoch 11/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0679 - acc: 0.9781\n",
      "Epoch 12/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0656 - acc: 0.9788\n",
      "Epoch 13/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0601 - acc: 0.9810\n",
      "Epoch 14/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0571 - acc: 0.9817\n",
      "Epoch 15/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0517 - acc: 0.9832\n",
      "Epoch 16/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0509 - acc: 0.9834\n",
      "Epoch 17/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0478 - acc: 0.9850\n",
      "Epoch 18/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0461 - acc: 0.9852\n",
      "Epoch 19/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0442 - acc: 0.9854\n",
      "Epoch 20/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0427 - acc: 0.9866\n",
      "20000/20000 [==============================] - 0s 14us/step\n",
      "40000/40000 [==============================] - 0s 7us/step\n",
      "Epoch 1/20\n",
      "40000/40000 [==============================] - 1s 27us/step - loss: 0.4694 - acc: 0.8670\n",
      "Epoch 2/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.2196 - acc: 0.9336\n",
      "Epoch 3/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1681 - acc: 0.9495\n",
      "Epoch 4/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1391 - acc: 0.9589\n",
      "Epoch 5/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1200 - acc: 0.9630\n",
      "Epoch 6/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1067 - acc: 0.9671\n",
      "Epoch 7/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0968 - acc: 0.9713\n",
      "Epoch 8/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0880 - acc: 0.9730\n",
      "Epoch 9/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0815 - acc: 0.9754\n",
      "Epoch 10/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0742 - acc: 0.9771\n",
      "Epoch 11/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0711 - acc: 0.9778\n",
      "Epoch 12/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0643 - acc: 0.9806\n",
      "Epoch 13/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0613 - acc: 0.9815\n",
      "Epoch 14/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0561 - acc: 0.9818\n",
      "Epoch 15/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0534 - acc: 0.9832\n",
      "Epoch 16/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0528 - acc: 0.9832\n",
      "Epoch 17/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0479 - acc: 0.9848\n",
      "Epoch 18/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0486 - acc: 0.9844\n",
      "Epoch 19/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0438 - acc: 0.9861\n",
      "Epoch 20/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0427 - acc: 0.9861\n",
      "20000/20000 [==============================] - 0s 14us/step\n",
      "40000/40000 [==============================] - 0s 7us/step\n",
      "Epoch 1/20\n",
      "40000/40000 [==============================] - 1s 27us/step - loss: 0.6830 - acc: 0.8003\n",
      "Epoch 2/20\n",
      "40000/40000 [==============================] - 1s 16us/step - loss: 0.3234 - acc: 0.9044\n",
      "Epoch 3/20\n",
      "40000/40000 [==============================] - 1s 16us/step - loss: 0.2456 - acc: 0.9269\n",
      "Epoch 4/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1982 - acc: 0.9407\n",
      "Epoch 5/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1627 - acc: 0.9506\n",
      "Epoch 6/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1422 - acc: 0.9561\n",
      "Epoch 7/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1251 - acc: 0.9618\n",
      "Epoch 8/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1134 - acc: 0.9650\n",
      "Epoch 9/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1037 - acc: 0.9677\n",
      "Epoch 10/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0957 - acc: 0.9705\n",
      "Epoch 11/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0891 - acc: 0.9730\n",
      "Epoch 12/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0832 - acc: 0.9745\n",
      "Epoch 13/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0760 - acc: 0.9768\n",
      "Epoch 14/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0727 - acc: 0.9778\n",
      "Epoch 15/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0681 - acc: 0.9782\n",
      "Epoch 16/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0628 - acc: 0.9803\n",
      "Epoch 17/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0613 - acc: 0.9801\n",
      "Epoch 18/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0572 - acc: 0.9824\n",
      "Epoch 19/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0549 - acc: 0.9821\n",
      "Epoch 20/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0521 - acc: 0.9834\n",
      "20000/20000 [==============================] - 0s 15us/step\n",
      "40000/40000 [==============================] - 0s 7us/step\n",
      "Epoch 1/20\n",
      "40000/40000 [==============================] - 1s 28us/step - loss: 0.6742 - acc: 0.8034\n",
      "Epoch 2/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.3142 - acc: 0.9081\n",
      "Epoch 3/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.2365 - acc: 0.9306\n",
      "Epoch 4/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1873 - acc: 0.9453\n",
      "Epoch 5/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1563 - acc: 0.9531\n",
      "Epoch 6/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1373 - acc: 0.9580\n",
      "Epoch 7/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1212 - acc: 0.9640\n",
      "Epoch 8/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1084 - acc: 0.9665\n",
      "Epoch 9/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1014 - acc: 0.9695\n",
      "Epoch 10/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0917 - acc: 0.9732\n",
      "Epoch 11/20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0838 - acc: 0.9739\n",
      "Epoch 12/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0788 - acc: 0.9749\n",
      "Epoch 13/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0720 - acc: 0.9779\n",
      "Epoch 14/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0685 - acc: 0.9793\n",
      "Epoch 15/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0666 - acc: 0.9797\n",
      "Epoch 16/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0618 - acc: 0.9807\n",
      "Epoch 17/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0578 - acc: 0.9819\n",
      "Epoch 18/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0557 - acc: 0.9825\n",
      "Epoch 19/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0535 - acc: 0.9831\n",
      "Epoch 20/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0482 - acc: 0.9848\n",
      "20000/20000 [==============================] - 0s 15us/step\n",
      "40000/40000 [==============================] - 0s 7us/step\n",
      "Epoch 1/20\n",
      "40000/40000 [==============================] - 1s 29us/step - loss: 0.6865 - acc: 0.8001\n",
      "Epoch 2/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.3089 - acc: 0.9106\n",
      "Epoch 3/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.2292 - acc: 0.9304\n",
      "Epoch 4/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1845 - acc: 0.9435\n",
      "Epoch 5/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1562 - acc: 0.9527\n",
      "Epoch 6/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1371 - acc: 0.9584\n",
      "Epoch 7/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1206 - acc: 0.9629\n",
      "Epoch 8/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.1098 - acc: 0.9667\n",
      "Epoch 9/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0990 - acc: 0.9699\n",
      "Epoch 10/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0912 - acc: 0.9718\n",
      "Epoch 11/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0863 - acc: 0.9740\n",
      "Epoch 12/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0792 - acc: 0.9757\n",
      "Epoch 13/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0721 - acc: 0.9782\n",
      "Epoch 14/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0696 - acc: 0.9782\n",
      "Epoch 15/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0661 - acc: 0.9796\n",
      "Epoch 16/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0621 - acc: 0.9810\n",
      "Epoch 17/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0588 - acc: 0.9813\n",
      "Epoch 18/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0541 - acc: 0.9831\n",
      "Epoch 19/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0513 - acc: 0.9834\n",
      "Epoch 20/20\n",
      "40000/40000 [==============================] - 1s 15us/step - loss: 0.0514 - acc: 0.9835\n",
      "20000/20000 [==============================] - 0s 16us/step\n",
      "40000/40000 [==============================] - 0s 7us/step\n",
      "Epoch 1/10\n",
      "40000/40000 [==============================] - 1s 22us/step - loss: 0.7656 - acc: 0.7926\n",
      "Epoch 2/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.3345 - acc: 0.9021\n",
      "Epoch 3/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.2633 - acc: 0.9225\n",
      "Epoch 4/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.2207 - acc: 0.9357\n",
      "Epoch 5/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1877 - acc: 0.9450\n",
      "Epoch 6/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1684 - acc: 0.9500\n",
      "Epoch 7/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1517 - acc: 0.9552\n",
      "Epoch 8/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1358 - acc: 0.9587\n",
      "Epoch 9/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1236 - acc: 0.9627\n",
      "Epoch 10/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1153 - acc: 0.9661\n",
      "20000/20000 [==============================] - 0s 14us/step\n",
      "40000/40000 [==============================] - 0s 4us/step\n",
      "Epoch 1/10\n",
      "40000/40000 [==============================] - 1s 22us/step - loss: 0.7640 - acc: 0.7905\n",
      "Epoch 2/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.3207 - acc: 0.9055\n",
      "Epoch 3/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.2455 - acc: 0.9276\n",
      "Epoch 4/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.2050 - acc: 0.9392\n",
      "Epoch 5/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1755 - acc: 0.9484\n",
      "Epoch 6/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1573 - acc: 0.9524\n",
      "Epoch 7/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1385 - acc: 0.9594\n",
      "Epoch 8/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1247 - acc: 0.9634\n",
      "Epoch 9/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1181 - acc: 0.9644\n",
      "Epoch 10/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1074 - acc: 0.9679\n",
      "20000/20000 [==============================] - 0s 15us/step\n",
      "40000/40000 [==============================] - 0s 4us/step\n",
      "Epoch 1/10\n",
      "40000/40000 [==============================] - 1s 23us/step - loss: 0.7858 - acc: 0.7867\n",
      "Epoch 2/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.3358 - acc: 0.9017\n",
      "Epoch 3/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.2603 - acc: 0.9250\n",
      "Epoch 4/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.2160 - acc: 0.9367\n",
      "Epoch 5/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1879 - acc: 0.9435\n",
      "Epoch 6/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1663 - acc: 0.9513\n",
      "Epoch 7/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1493 - acc: 0.9552\n",
      "Epoch 8/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1363 - acc: 0.9591\n",
      "Epoch 9/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1262 - acc: 0.9609\n",
      "Epoch 10/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1147 - acc: 0.9653\n",
      "20000/20000 [==============================] - 0s 16us/step\n",
      "40000/40000 [==============================] - 0s 4us/step\n",
      "Epoch 1/10\n",
      "40000/40000 [==============================] - 1s 23us/step - loss: 1.1193 - acc: 0.6932\n",
      "Epoch 2/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.4829 - acc: 0.8573\n",
      "Epoch 3/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.3785 - acc: 0.8885\n",
      "Epoch 4/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.3208 - acc: 0.9062\n",
      "Epoch 5/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.2811 - acc: 0.9167\n",
      "Epoch 6/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.2472 - acc: 0.9263\n",
      "Epoch 7/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.2200 - acc: 0.9362\n",
      "Epoch 8/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1963 - acc: 0.9422\n",
      "Epoch 9/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1772 - acc: 0.9475\n",
      "Epoch 10/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1611 - acc: 0.9519\n",
      "20000/20000 [==============================] - 0s 16us/step\n",
      "40000/40000 [==============================] - 0s 4us/step\n",
      "Epoch 1/10\n",
      "40000/40000 [==============================] - 1s 24us/step - loss: 1.1189 - acc: 0.6828\n",
      "Epoch 2/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.4867 - acc: 0.8556\n",
      "Epoch 3/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.3713 - acc: 0.8919\n",
      "Epoch 4/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.3125 - acc: 0.9090\n",
      "Epoch 5/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.2748 - acc: 0.9214\n",
      "Epoch 6/10\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.2443 - acc: 0.9285\n",
      "Epoch 7/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.2141 - acc: 0.9383\n",
      "Epoch 8/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1919 - acc: 0.9445\n",
      "Epoch 9/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1767 - acc: 0.9485\n",
      "Epoch 10/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1617 - acc: 0.9528\n",
      "20000/20000 [==============================] - 0s 17us/step\n",
      "40000/40000 [==============================] - 0s 4us/step\n",
      "Epoch 1/10\n",
      "40000/40000 [==============================] - 1s 25us/step - loss: 1.0997 - acc: 0.7128\n",
      "Epoch 2/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.4652 - acc: 0.8638\n",
      "Epoch 3/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.3716 - acc: 0.8919\n",
      "Epoch 4/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.3165 - acc: 0.9072\n",
      "Epoch 5/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.2735 - acc: 0.9199\n",
      "Epoch 6/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.2381 - acc: 0.9299\n",
      "Epoch 7/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.2094 - acc: 0.9380\n",
      "Epoch 8/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1863 - acc: 0.9442\n",
      "Epoch 9/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1693 - acc: 0.9491\n",
      "Epoch 10/10\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1549 - acc: 0.9538\n",
      "20000/20000 [==============================] - 0s 17us/step\n",
      "40000/40000 [==============================] - 0s 4us/step\n",
      "Epoch 1/20\n",
      "40000/40000 [==============================] - 1s 26us/step - loss: 0.7406 - acc: 0.7936\n",
      "Epoch 2/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.3331 - acc: 0.9019\n",
      "Epoch 3/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.2623 - acc: 0.9229\n",
      "Epoch 4/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.2174 - acc: 0.9372\n",
      "Epoch 5/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1876 - acc: 0.9436\n",
      "Epoch 6/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1659 - acc: 0.9498\n",
      "Epoch 7/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1493 - acc: 0.9559\n",
      "Epoch 8/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1358 - acc: 0.9602\n",
      "Epoch 9/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1244 - acc: 0.9624\n",
      "Epoch 10/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1135 - acc: 0.9655\n",
      "Epoch 11/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1063 - acc: 0.9683\n",
      "Epoch 12/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.0963 - acc: 0.9711\n",
      "Epoch 13/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.0924 - acc: 0.9720\n",
      "Epoch 14/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.0897 - acc: 0.9732\n",
      "Epoch 15/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.0811 - acc: 0.9752\n",
      "Epoch 16/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.0798 - acc: 0.9750\n",
      "Epoch 17/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.0733 - acc: 0.9770\n",
      "Epoch 18/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.0695 - acc: 0.9789\n",
      "Epoch 19/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.0652 - acc: 0.9798\n",
      "Epoch 20/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.0622 - acc: 0.9815\n",
      "20000/20000 [==============================] - 0s 18us/step\n",
      "40000/40000 [==============================] - 0s 4us/step\n",
      "Epoch 1/20\n",
      "40000/40000 [==============================] - 1s 27us/step - loss: 0.7349 - acc: 0.8012\n",
      "Epoch 2/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.3201 - acc: 0.9060\n",
      "Epoch 3/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.2541 - acc: 0.9257\n",
      "Epoch 4/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.2149 - acc: 0.9375\n",
      "Epoch 5/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1894 - acc: 0.9443\n",
      "Epoch 6/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1694 - acc: 0.9496\n",
      "Epoch 7/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1552 - acc: 0.9541\n",
      "Epoch 8/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1421 - acc: 0.9580\n",
      "Epoch 9/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1307 - acc: 0.9605\n",
      "Epoch 10/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1200 - acc: 0.9642\n",
      "Epoch 11/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1110 - acc: 0.9663\n",
      "Epoch 12/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1041 - acc: 0.9681\n",
      "Epoch 13/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.0971 - acc: 0.9698\n",
      "Epoch 14/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.0943 - acc: 0.9713\n",
      "Epoch 15/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.0864 - acc: 0.9733\n",
      "Epoch 16/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.0809 - acc: 0.9748\n",
      "Epoch 17/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.0776 - acc: 0.9762\n",
      "Epoch 18/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.0745 - acc: 0.9762\n",
      "Epoch 19/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.0712 - acc: 0.9777\n",
      "Epoch 20/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.0654 - acc: 0.9793\n",
      "20000/20000 [==============================] - 0s 18us/step\n",
      "40000/40000 [==============================] - 0s 4us/step\n",
      "Epoch 1/20\n",
      "40000/40000 [==============================] - 1s 27us/step - loss: 0.7435 - acc: 0.8001\n",
      "Epoch 2/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.3335 - acc: 0.9029\n",
      "Epoch 3/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.2555 - acc: 0.9254\n",
      "Epoch 4/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.2168 - acc: 0.9359\n",
      "Epoch 5/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1872 - acc: 0.9458\n",
      "Epoch 6/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1640 - acc: 0.9514\n",
      "Epoch 7/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1452 - acc: 0.9578\n",
      "Epoch 8/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1313 - acc: 0.9607\n",
      "Epoch 9/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1193 - acc: 0.9644\n",
      "Epoch 10/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1104 - acc: 0.9670\n",
      "Epoch 11/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1024 - acc: 0.9691\n",
      "Epoch 12/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.0955 - acc: 0.9715\n",
      "Epoch 13/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.0881 - acc: 0.9734\n",
      "Epoch 14/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.0815 - acc: 0.9756\n",
      "Epoch 15/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.0772 - acc: 0.9764\n",
      "Epoch 16/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.0745 - acc: 0.9780\n",
      "Epoch 17/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.0681 - acc: 0.9788\n",
      "Epoch 18/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.0649 - acc: 0.9806\n",
      "Epoch 19/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.0640 - acc: 0.9800\n",
      "Epoch 20/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.0604 - acc: 0.9813\n",
      "20000/20000 [==============================] - 0s 19us/step\n",
      "40000/40000 [==============================] - 0s 4us/step\n",
      "Epoch 1/20\n",
      "40000/40000 [==============================] - 1s 28us/step - loss: 1.1126 - acc: 0.6943\n",
      "Epoch 2/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.4792 - acc: 0.8568\n",
      "Epoch 3/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.3712 - acc: 0.8910\n",
      "Epoch 4/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.3155 - acc: 0.9068\n",
      "Epoch 5/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.2746 - acc: 0.9196\n",
      "Epoch 6/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.2392 - acc: 0.9293\n",
      "Epoch 7/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.2126 - acc: 0.9363\n",
      "Epoch 8/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1923 - acc: 0.9437\n",
      "Epoch 9/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1706 - acc: 0.9489\n",
      "Epoch 10/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1591 - acc: 0.9530\n",
      "Epoch 11/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1462 - acc: 0.9560\n",
      "Epoch 12/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1342 - acc: 0.9593\n",
      "Epoch 13/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1272 - acc: 0.9609\n",
      "Epoch 14/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1188 - acc: 0.9639\n",
      "Epoch 15/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1126 - acc: 0.9660\n",
      "Epoch 16/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1072 - acc: 0.9673\n",
      "Epoch 17/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1016 - acc: 0.9692\n",
      "Epoch 18/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.0971 - acc: 0.9697\n",
      "Epoch 19/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.0901 - acc: 0.9719\n",
      "Epoch 20/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.0869 - acc: 0.9729\n",
      "20000/20000 [==============================] - 0s 19us/step\n",
      "40000/40000 [==============================] - 0s 4us/step\n",
      "Epoch 1/20\n",
      "40000/40000 [==============================] - 1s 28us/step - loss: 1.0906 - acc: 0.7216\n",
      "Epoch 2/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.4527 - acc: 0.8658\n",
      "Epoch 3/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.3559 - acc: 0.8956\n",
      "Epoch 4/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.3055 - acc: 0.9109\n",
      "Epoch 5/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.2642 - acc: 0.9219\n",
      "Epoch 6/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.2333 - acc: 0.9317\n",
      "Epoch 7/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.2057 - acc: 0.9386\n",
      "Epoch 8/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1858 - acc: 0.9462\n",
      "Epoch 9/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1674 - acc: 0.9501\n",
      "Epoch 10/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1514 - acc: 0.9548\n",
      "Epoch 11/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1410 - acc: 0.9571\n",
      "Epoch 12/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1303 - acc: 0.9607\n",
      "Epoch 13/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1193 - acc: 0.9643\n",
      "Epoch 14/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1131 - acc: 0.9649\n",
      "Epoch 15/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1054 - acc: 0.9680\n",
      "Epoch 16/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.0999 - acc: 0.9696\n",
      "Epoch 17/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.0937 - acc: 0.9712\n",
      "Epoch 18/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.0895 - acc: 0.9731\n",
      "Epoch 19/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.0830 - acc: 0.9758\n",
      "Epoch 20/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.0790 - acc: 0.9760\n",
      "20000/20000 [==============================] - 0s 20us/step\n",
      "40000/40000 [==============================] - 0s 4us/step\n",
      "Epoch 1/20\n",
      "40000/40000 [==============================] - 1s 29us/step - loss: 1.1233 - acc: 0.6955\n",
      "Epoch 2/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.4793 - acc: 0.8588\n",
      "Epoch 3/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.3751 - acc: 0.8898\n",
      "Epoch 4/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.3203 - acc: 0.9069\n",
      "Epoch 5/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.2760 - acc: 0.9196\n",
      "Epoch 6/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.2422 - acc: 0.9286\n",
      "Epoch 7/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.2155 - acc: 0.9363\n",
      "Epoch 8/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1910 - acc: 0.9437\n",
      "Epoch 9/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1718 - acc: 0.9489\n",
      "Epoch 10/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1584 - acc: 0.9530\n",
      "Epoch 11/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1435 - acc: 0.9570\n",
      "Epoch 12/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1326 - acc: 0.9603\n",
      "Epoch 13/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1244 - acc: 0.9615\n",
      "Epoch 14/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1171 - acc: 0.9644\n",
      "Epoch 15/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1093 - acc: 0.9670\n",
      "Epoch 16/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.1047 - acc: 0.9692\n",
      "Epoch 17/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.0984 - acc: 0.9698\n",
      "Epoch 18/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.0910 - acc: 0.9720\n",
      "Epoch 19/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.0860 - acc: 0.9734\n",
      "Epoch 20/20\n",
      "40000/40000 [==============================] - 0s 8us/step - loss: 0.0827 - acc: 0.9748\n",
      "20000/20000 [==============================] - 0s 21us/step\n",
      "40000/40000 [==============================] - 0s 4us/step\n",
      "Epoch 1/20\n",
      "60000/60000 [==============================] - 2s 31us/step - loss: 0.4007 - acc: 0.8851\n",
      "Epoch 2/20\n",
      "60000/60000 [==============================] - 1s 17us/step - loss: 0.1892 - acc: 0.9439\n",
      "Epoch 3/20\n",
      "60000/60000 [==============================] - 1s 17us/step - loss: 0.1432 - acc: 0.9567\n",
      "Epoch 4/20\n",
      "60000/60000 [==============================] - 1s 17us/step - loss: 0.1185 - acc: 0.9643\n",
      "Epoch 5/20\n",
      "60000/60000 [==============================] - 1s 17us/step - loss: 0.1050 - acc: 0.9678\n",
      "Epoch 6/20\n",
      "60000/60000 [==============================] - 1s 17us/step - loss: 0.0929 - acc: 0.9715\n",
      "Epoch 7/20\n",
      "60000/60000 [==============================] - 1s 17us/step - loss: 0.0848 - acc: 0.9737\n",
      "Epoch 8/20\n",
      "60000/60000 [==============================] - 1s 17us/step - loss: 0.0775 - acc: 0.9764\n",
      "Epoch 9/20\n",
      "60000/60000 [==============================] - 1s 17us/step - loss: 0.0723 - acc: 0.9780\n",
      "Epoch 10/20\n",
      "60000/60000 [==============================] - 1s 17us/step - loss: 0.0693 - acc: 0.9785\n",
      "Epoch 11/20\n",
      "60000/60000 [==============================] - 1s 17us/step - loss: 0.0637 - acc: 0.9802\n",
      "Epoch 12/20\n",
      "60000/60000 [==============================] - 1s 17us/step - loss: 0.0602 - acc: 0.9814\n",
      "Epoch 13/20\n",
      "60000/60000 [==============================] - 1s 17us/step - loss: 0.0585 - acc: 0.9816\n",
      "Epoch 14/20\n",
      "60000/60000 [==============================] - 1s 17us/step - loss: 0.0546 - acc: 0.9827\n",
      "Epoch 15/20\n",
      "60000/60000 [==============================] - 1s 18us/step - loss: 0.0512 - acc: 0.9834\n",
      "Epoch 16/20\n",
      "60000/60000 [==============================] - 1s 18us/step - loss: 0.0501 - acc: 0.9841\n",
      "Epoch 17/20\n",
      "60000/60000 [==============================] - 1s 17us/step - loss: 0.0481 - acc: 0.9844\n",
      "Epoch 18/20\n",
      "60000/60000 [==============================] - 1s 17us/step - loss: 0.0456 - acc: 0.9860\n",
      "Epoch 19/20\n",
      "60000/60000 [==============================] - 1s 17us/step - loss: 0.0444 - acc: 0.9857\n",
      "Epoch 20/20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "60000/60000 [==============================] - 1s 17us/step - loss: 0.0439 - acc: 0.9861\n",
      "CPU times: user 7min 56s, sys: 1min 6s, total: 9min 3s\n",
      "Wall time: 3min 43s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# fix random seed for reproducibility (this might work or might not work \n",
    "# depending on each library's implenentation)\n",
    "seed = 7\n",
    "numpy.random.seed(seed)\n",
    "\n",
    "# create the sklearn model for the network\n",
    "model_init_batch_epoch_CV = KerasClassifier(build_fn=create_model_2, verbose=1)\n",
    "\n",
    "# we choose the initializers that came at the top in our previous cross-validation!!\n",
    "init_mode = ['glorot_uniform', 'uniform'] \n",
    "batches = [128, 512]\n",
    "epochs = [10, 20]\n",
    "\n",
    "# grid search for initializer, batch size and number of epochs\n",
    "param_grid = dict(epochs=epochs, batch_size=batches, init=init_mode)\n",
    "grid = GridSearchCV(estimator=model_init_batch_epoch_CV, \n",
    "                    param_grid=param_grid,\n",
    "                    cv=3)\n",
    "grid_result = grid.fit(x_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best Accuracy for 0.9712 using {'batch_size': 128, 'epochs': 20, 'init': 'glorot_uniform'}\n",
      "mean=0.9687, std=0.002174 using {'batch_size': 128, 'epochs': 10, 'init': 'glorot_uniform'}\n",
      "mean=0.966, std=0.000827 using {'batch_size': 128, 'epochs': 10, 'init': 'uniform'}\n",
      "mean=0.9712, std=0.0006276 using {'batch_size': 128, 'epochs': 20, 'init': 'glorot_uniform'}\n",
      "mean=0.97, std=0.001214 using {'batch_size': 128, 'epochs': 20, 'init': 'uniform'}\n",
      "mean=0.9594, std=0.001476 using {'batch_size': 512, 'epochs': 10, 'init': 'glorot_uniform'}\n",
      "mean=0.9516, std=0.003239 using {'batch_size': 512, 'epochs': 10, 'init': 'uniform'}\n",
      "mean=0.9684, std=0.003607 using {'batch_size': 512, 'epochs': 20, 'init': 'glorot_uniform'}\n",
      "mean=0.9633, std=0.0007962 using {'batch_size': 512, 'epochs': 20, 'init': 'uniform'}\n"
     ]
    }
   ],
   "source": [
    "# print results\n",
    "print(f'Best Accuracy for {grid_result.best_score_:.4} using {grid_result.best_params_}')\n",
    "means = grid_result.cv_results_['mean_test_score']\n",
    "stds = grid_result.cv_results_['std_test_score']\n",
    "params = grid_result.cv_results_['params']\n",
    "for mean, stdev, param in zip(means, stds, params):\n",
    "    print(f'mean={mean:.4}, std={stdev:.4} using {param}')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
